NASA SP-6101 (11) 


ISSUES IN 
NASA PROGRAM 
AND PROJECT 
MANAGEMENT 

A Collection of Papers on 
Aerospace Management Issues 


edited by 

Edward J. Hoffman 

Program Manager 

NASA Program and Project Management Initiative 

William M. Lawbaugh 

Technical and Administrative Services Corporation 
(TADCORPS) 



National Aeronautics and Space Administration 
Office of Management Systems and Facilities 
Scientific and Technical Information Program 
Washington, DC 1996 



Issues in NASA Program and Project Management 

A Collection of Papers on Aerospace Issues 


National Aeronautics and Space Administration 


Autumn 1996 


Page 

1 NASA’s Project Management Development Process Dr. Edward J. Hoffman 

The PPMI program manager explains NASA’s newest opportunity for career development. The Project 
Management Development Process (PMDP), based upon thorough research and careful listening, is the Agency’s 
first systematic process for the development, growth and improvement of program and project managers. 

7 Better Decisions through Structured Analysis Morgan D. Jones 

A longtime analyst of Soviet space programs has discovered, tested and described 14 ways to “overcome the sub- 
jective tendencies of the human mind” for making better, smarter decisions. He outlines two of them here: 
devil’s advocacy and utility analysis. Both are from his new book. The Thinkers Toolkit. 

11 TechTracS : NASA’s Commercial Technology Management System Kevin Barquinero & 

Douglas Cannon 

Specialists from the NASA commercial technology management team in the Office of Space Access and 
Technology detail the four stages of NASA’s latest commercial technology management system. The 
Commercial Technology Division manages a multimedia technology utilization program to identify, market, 
encourage and track the technology transfer from the government to private industry. 

15 Today’s Management Techniques and Tools Ernest M. Hahne 

A veteran consultant in systems engineering and training asks: “Are we missing something?” In an age of 
increasing complexity and layers of Federal requirements, the “gurus of management science” may be over- 
looking a primary risk analysis tool. Hahne offers several axioms and lessons learned from his extensive expe- 
rience. 

33 Program Control in NASA: Needs and Opportunities William E. Lilly 

The longtime comptroller of NASA leads a study team at the National Academy of Public Administration in 
evaluation of common program control processes. He develops a model of program control functions and offers 
several recommendations on program control needs and opportunities in NASA. 

47 Resources for NASA Managers Dr. William M. Lawbaugh 

Book and video reviews of interest to program and project managers. 


SP-6101 (1 1 ) Issues in NASA Program and Project Management is tenth in a series from NASA’s Program and Project 
Management Initiative. This series is collected and edited by Dr. Edward J. Hoffman with Dr. William M. Lawbaugh. 
Statements and opinions are those of the authors and do not represent official policy of NASA or the U.S. Government. 
Useful and enlightening material is welcome, and diversity of ideas is encouraged. 


Inquiries should be directed to Dr. Edward J. Hoffman , Program Manager Office of Training 
and Development , Code FT, NASA Headquarters ; Washington, DC 20546-0001 . 

Email: ed.hojfman@hq.nasa.gov or wlawbaugh@ tadcorps.com 


1 




NASA's Project Management Development Process 


NASA’s Project Management Development Process 


by Dr. Edward J. Hoffman 


Professional development has always been a major 
concern at NASA, but until recently there has been 
no systematic process for the development, growth 
and improvement of the Agency’s people working on 
projects. 

Ten years ago NASA established the Program 
Project Management Initiative (PPMI) to provide 
project management development in advance of 
need. Over the years, PPMI has met the needs of 
thousands of NASA employees. In recent years 
greater emphasis has been placed on having a more 
systematic. Agencywide process for the development 
of people in projects. 

In order to establish a systematic process which 
would best represent NASA and meet the demands 
of our workforce, a study was conducted to deter- 
mine the components of an effective development 
process. The researchers interviewed over 1 50 peo- 
ple from five NASA Centers at various stages of 
their careers. The central finding of this study was 
the need for a NASA project management develop- 
ment process that would be voluntary, nonbureacrat- 
ic, open to many, and involve a minimum of paper- 
work. 

In August of 1993 the results of the full-scale Career 
Development Research Study were published. The 
study, titled Career Development for Project 
Management, was designed to create an empirically 
based foundation for the Project Management 
Development Process (PMDP). A shortened version 
of this document was published in the Winter 1994 
edition of Issues in NASA Program and Project 
Management, NASA SP-6 10 1 (08). 

Out of this significant study came the four-level pro- 
fessional development chart (Figure 1) for people in 
projects, which has been widely reprinted. More 
directly, this study led to NASA senior manage- 
ment’s recommendation to establish and institution- 


alize the Project Management Development Process. 
The PMDP, formally established in 1995, has 
received a great deal of interest from other govern- 
ment agencies and industry, as well as international 
organizations. 

While communicating the PMDP and receiving ideas 
for implementation, General Spence “Sam” 
Armstrong, Associate Administrator for Human 
Resources and Education, and I visited each and 
every NASA Center. What we found was a great 
depth of concern for fairness and equity in the devel- 
opment of program and project managers. Loud and 
clear, the predominant concerns were that NASA 
management would openly support and communi- 
cate such a process for NASA employees, and that 
the PMDP not result in a system which forced Center 
managers to hire project managers who have only 
gained “checklist” qualification. It should be neither 
a barrier nor a guarantee of promotion, but rather a 
wide-open, professional enhancement opportunity. 

In January of 1995 a group of NASA project man- 
agers and human resources management profession- 
als met with PPMI staff at Kennedy Space Center to 
plan the launch of PMDP. This group refined and 
expanded the developmental experience and training 
recommendations described within the two PMDP 
handbooks. (Both the participant and supervisors 
PMDP handbooks are available through local train- 
ing offices, as well as through the NASA PPMI web 
site: http://www.hq.nasa.gov/office/HR-Education/ 
training/ppmi.htm.) The group also refined the 
PMDP structure found in Figure 1, showing the 
preparation and four levels of accomplishment, plus 
the reviews needed for advancement to ongoing pro- 
fessional development. 

Note that PMDP is a process, not a program or prod- 
uct, and it is open to senior managers as well as to 
new hires and mid-career NASA employees. The 
development process is ongoing. It is also important 
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to keep in mind that PMDP is not a training process; 
rather, it is a development process for enhancing crit- 
ical competencies associated with the work of people 
in projects. With this direction, PMDP encourages 
gaining competency through appropriate and specif- 
ic work assignments which are supplemented by 
well-timed training and development rotations and 
assignments. 

The Project Management Development Process 
became official on August 22, 1995, when General 
John R. Dailey, Acting Deputy Administrator, noti- 
fied officials at headquarters and directors of field 
installations. In part he wrote: 

“During this time of dramatic change at NASA, it is 
critical to reemphasize the importance of our career 


development programs for our employees. We must 
maintain our commitment to the future by supporting 
the ongoing learning and necessary work and train- 
ing experiences of the NASA workforce. It is the 
responsibility of each Installation to support mem- 
bers of the project management community in 
receiving the proper experiences during their career. 

“Over the past two years, we have instituted many 
efforts within NASA to codify a more consistent 
approach for managing our projects. As part of these 
efforts, the Program Management Council has been 
established to review major programs, and NMI 
7120.4 (Management of Major System Programs and 
Projects) has been enhanced to better reflect the cri- 
teria for the effective management of our major pro- 
grams and projects. 
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“To successfully implement these changes, we must 
communicate and institutionalize these project man- 
agement standards to make them a natural compo- 
nent of how we do our work. A key vehicle for 
assisting this integration is the PMDP. The PMDP is 
an Agencywide professional development process 
for employees interested in project management. It 
will assist employees with identifying work and 
training experiences beneficial to their professional 
development. In addition, it will also support a more 
consistent application of successful project manage- 
ment practices across NASA. There are significant 
benefits for both NASA and our employees.” 

Shortly after General Dailey’s announcement, mate- 
rials describing the process and startup procedures 
were sent to the training and development office and 
to senior management at each NASA installation. 
Included were a 14-minute videotape, guidebooks 
for both participants and their mentor or supervisor 
and an Individual Development Plan (IDP) advisory, 
listing potential ways to gain specific skills and 
experiences. 

PMDP Guidelines 

To support the PMDP, a participant handbook and a 
supervisor handbook were created. Both guidebooks 
begin with a description of the Project Management 
Development Process, including the three main goals 
of strengthening “the consistent application of suc- 
cessful project management practices” across all of 
NASA, providing “clear information” about profes- 
sional development opportunities in the Agency, and 
identifying “work experiences, training and develop- 
mental assignments” which enable people in projects 
to enhance their competencies and support career 
development goals. The PMDP is not a selection 
process that limits future selection, nor is it a guar- 
antee of future promotion. It is intended to support 
the enhancement of professional capabilities and 
growth, not to preselect future managers or limit pro- 
motion opportunities to a select few. 

PMDP is a “tool” to assist the NASA employee in 
development planning. Its starting point and frame- 
work is the employee’s Individual Development 
Plan, a projection of the applicant’s career objectives 


along with the on-the-job work requirements to be 
supplemented by training and work assignments. 
The IDP is worked out in conjunction with a super- 
visor and, if desired, a mentor who will guide the 
employee through the PMDP. An IDP can be a 
Center-specific process, the NASA process or an 
approach the individual and immediate supervisor 
support. 

The mentor and/or supervisor will ordinarily begin 
the process by asking the applicant to fill out a 
Record of Accomplishment (RoA), which will help 
each of them to determine where the employee is in 
terms of professional development. The RoA is a 
simple form, not unlike a resume or curriculum vitae, 
listing relevant education, work experience, training 
courses and other accomplished development oppor- 
tunities. There is a sample RoA page within the par- 
ticipant handbook. It is the development of this RoA 
or listing of work and educational experiences that 
provides the individual and the supervisor with the 
information necessary to discuss an appropriate ini- 
tial level of entry into the PMDR More important 
than the initial level, however, is the fact that the 
RoA forces a person to take the time to document 
specifically the past experiences which establish a 
skill mark and visually supports the person in plan- 
ning for future goals. 

During the initial planning process the supervisor 
should provide honest feedback about the employ- 
ee’s accomplishments, skills and areas of growth. In 
addition, a mentor may be extremely useful for pro- 
viding guidance of expertise which the supervisor 
might not have. It is important to keep in mind that 
the intent of this process is not to ascend to the high- 
est level possible; the objective is to document the 
experiences gained to date honestly and clarify indi- 
vidual competencies, areas for growth and specific 
steps for enhancing competency. The PMDP should 
open up a window of needs and concurrent opportu- 
nities for gaining competencies. 

Both the RoA and the IDP are simply professional 
development tools. Completion of the forms is not 
an end in itself nor a contract for advancement. 
Emphasis should always be on the development 
process, not merely filling out forms and getting 


3 



them signed. (In fact, the intent is that after the ini- 
tial establishment of an RoA and a IDP, documenta- 
tion and maintenance become simple.) 

To assist the employee in building the Individual 
Development Plan, the PPMI has created the IDP 
Advisor, a 34-page handbook with locator that pro- 
vides specific examples of potential work experi- 
ences and training recommendations for each of the 
four levels. The IDP Advisor is intended as a cata- 
lyst of potential activities, not a prescription. It will 
also be periodically updated to reflect management 
changes and offer new ideas. 

Management Development Process 

At the heart of PMDP are the core competencies for 
each of the four levels. Within each level are knowl- 
edge, skills and competencies clustered within the 
following eight general factors: 

• organizational knowledge 

• technical knowledge 

• technical management 

• project life cycle and program control 

• contract acquisition 

• individual and team development 

• Agency, business and international relations 

• risk management and safety 

As can be seen in the four-level competencies chart, 
hands-on technical expertise is emphasized in Level 
One, while broad leadership competencies are 
emphasized increasingly up through Level Four. 

Typically, Level One is considered entry level after 
one to three years of basic discipline development 
and work experience. It focuses on hands-on engi- 
neering tasks. One critical component is under- 
standing NASA guidance on the management of pro- 


jects as documented in the soon to be released NPD 
7120.4 (and concurrent handbook). Also considered 
critical to job performance in terms of organization- 
al knowledge is some kind of understanding of and 
experience with the NASA Project Life Cycle. The 
supervisor and/or mentor may specify the observa- 
tion of at least one program review per phase local- 
ly, plus several observations of project life cycle 
reviews with the Center director or directorate. 

In the technical area, hands-on hardware/software 
operations are deemed critical, along with configura- 
tion management systems and procedures, plus qual- 
ity assurance. Over a period of three or four years, 
the entry-level candidate is expected to develop thor- 
ough technical knowledge in his or her discipline, 
and participate in both operations analysis and 
research activities. 

Three core training experiences are required for 
Level One candidates, each related to a correspond- 
ing work requirement. The Program Control 
Overview course relates to Project Life Cycle devel- 
opment activities, while Systems Engineering and 
Task Management enrich the technical program flow 
as well as cost and scheduling work requirements. 
(Be aware that the Task Management course is called 
by other titles at local Center offerings, typically 
Project Leadership Simulation.) Several other PPMI 
courses are encouraged, depending upon the candi- 
date’s work schedule and experiences. The funda- 
mental idea is to make theory and practice mutually 
beneficial. As the one informs the other, the candi- 
date in Level One obtains a broad foundation of 
knowledge and experience necessary for systematic 
career development. 

Level Two candidates, on the other hand, typically 
find themselves gaining valuable experience as a 
technical expert or as a leader on small subsystems 
or instrumentation projects. Their required courses 
are Project Management and Program Control 
Overview. At this point the candidate should be 
designing, developing, testing and reviewing hard- 
ware/software at the test bed and system level. He or 
she may serve as the leader of a matrixed team, and 
lead team meetings. 
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Their knowledge of issues in interagency and inter- 
national relations can be enhanced through work 
assignments, task teams and possible rotations. The 
candidates should by now be writing reports, 
requirements and Statements of Work (SOWs) for a 
subsystem. At this level they are encouraged to 
enhance their managerial skills because they will be 
assuming more managerial duties, with their com- 
munication and interpersonal skills becoming more 
important. 

Level Three reflects a systems manager perspective. 
A candidate is expected to manage a systems-level 
project, including contractors and NASA team mem- 
bers. The individual manager is usually responsible 
for contract management, developing and monitor- 
ing master schedules, maintaining budget control, 
preparing a Program Operating Plan (POP) and man- 
aging the overall system life cycle. The project man- 
ager at this level is seen as an Agency resource and 
may be asked to serve NASA-wide boards. 

The Advanced Project Management course must be 
completed before completing Level Three. Courses 
in management and Performance Measurement 
Systems are encouraged, along with the Project 
Management Shared Experiences Program offered 
every other year. 

The PMSEP is also encouraged for participants in 
Level Four, along with the Senior Executive 
Program. Work or developmental experiences 
require knowledge for NASA’s political environment 
and strategic planning, as on Level Three. 

At Level Four people are expected to interface with 
all project implementation organizations internal to 
NASA (Mission Assurance, Engineering, Opera- 
tions, Acquisition) and external organizations (indus- 
try, academia, international partners, and U.S. gov- 
erning bodies). They are expected to manage and be 
held accountable for the entire program or project 
which they are leading. 

In terms of individual and team development activi- 
ties, Level Four people should become adept in man- 
aging people (including recruitment, human resource 


development, coaching, mentoring and personnel 
evaluation) and teamwork (including team selection, 
motivation, rewards, empowerment and conflict res- 
olution). They will be known for their decision mak- 
ing skills, creative problem solving and trou- 
bleshooting experiences. Working across Agency, 
Center and international lines, they learn to deal with 
other cultures and handle external factors which act 
on any project. 

Of course, not everyone who chooses to enter the 
PMDP process will want to move through all four 
levels. Our NASA history and past practices show 
many talented, successful scientists and engineers 
have found their niche, working on technical tasks, 
managing small projects or balancing laboratory 
work and management. 

Others will choose to progress through the ranks and 
up the four levels of accomplishment. They will 
enter the PMDP, meet with their mentor and/or 
supervisor regularly, plan their training and profes- 
sional development activities, discuss their IDP and 
document their progress in their RoA, and make 
adjustments to the IDP at least annually, until the 
career objectives are fully achieved. 

Moving from one level of achievement to the next 
higher one also involves a minimum of paperwork 
and procedure. First of all, a candidate’s supervisor 
has the authority to recommend individual place- 
ment up to Level Two. To begin Level Three, how- 
ever, the supervisor will have to submit a completed 
IDP or RoA from Level Two to the Installation PMC 
Panel for review and approval. Level Four entry 
requires the same procedure, plus concurrence from 
the Center Director and the Agency-level PMC. In 
each case, the Center’s human resource organization 
will receive a copy of the revised IDP and complet- 
ed RoA. Upon completion of each level the individ- 
ual will receive an Agency certificate of recognition. 
For this to happen, the interested candidate must 
make sure that the local human resource department 
has forwarded the candidate’s name to the NASA 
Office of Training & Development. Candidates with 
questions about this can call this office at 
202 - 358 - 0300 . 
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The primary responsibility for professional develop- 
ment rests with the applicant. We tried to keep the 
PMDP process as self-directed and self-monitored as 
possible, with plenty of assistance from mentors, 
supervisors and the PPMI. We have developed 
PMDP handbooks for the supervisor/mentor as well 
as the participant, and we ask that the applicants 
themselves who choose their mentors, if any, com- 
plete and process the documentation such as their 
own IDP and RoA, and that they schedule all meet- 
ings with mentor and/or supervisor for guidance and 
feedback. 

The Installation Panels at the third and fourth levels 
set policy for the Center’s PMDP to ensure fair and 
consistent treatment for all participants. They also 
approve “graduation” to the next level of accom- 
plishment. 


Likewise, the Program Management Council sets 
policy Agencywide for the PMDP and approves 
entry into Level Four. Our Headquarters PPMI 
Office coordinates the Agencywide project manage- 
ment training program in support of the PMDP and 
continues to periodically evaluate the PMDP as it 
relates to the quality of project management within 
NASA. With all this attention at various levels of 
NASA management, constant revision, upgrading 
and improvement of the PMDP process is expected 
as conditions change and as the needs of the Agency 
evolve. 

With the increasing emphasis around the world on 
core competencies for project managers, the PMDP 
provides unlimited opportunities for NASA project 
managers to plan and manage their own future. 
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Better Decisions Through Structured Analysis: 
Overcoming the Subjective Tendencies of the 
Human Mind 

by Morgan D. Jones 


As a Central Intelligence Agency analyst of Soviet 
space programs in the late Sixties and early 
Seventies, I was constantly challenged to estimate 
the capabilities and intentions — past, present, and 
future — of these programs. I believe a fair review of 
my work in those years would show that most of my 
analytic judgments were on the money. But on those 
(dare I say “rare”) occasions when I erred, as we 
humans are prone, I would review my analysis to see 
where I had gone wrong. Invariably I discovered 
that, for whatever reason, I had given insufficient 
consideration or weight to the alternative course of 
action which the Soviets had chosen. 

I may have estimated, for solid and justifiable rea- 
sons, that a certain Soviet program would move in a 
particular direction . . . and it didn’t. Or I may have 
estimated a program would not move in a particular 
direction . . . and it did. As we all know, one learns 
little from being right, and volumes from being 
wrong. And what I learned from my “rare,” always 
galling, analytic failures was that, despite my keen- 
est efforts, I had not been objective in my analysis. 

Do a quick exercise with me. Think of someone 
with whom you work closely every day. 

Now visualize that person’s face and recall the 
last time you spoke with him or her. 

Now imagine that you read a newspaper article 
alleging this person has embezzled a great deal of 
money from your organization. 

What is your instant reaction? 

You immediately formed an opinion, didn’t you? 
“That person is incapable of stealing?” Or, “Yeah, 


that person could be an embezzler. ” Or something 
else. 

Have you ever wondered why we humans impul- 
sively take sides on issues? Why can’t we approach 
problems objectively, without instantly harboring an 
opinion about them? The answer, provided by cogni- 
tive science, is that the human mind is programmed 
to be opinionated, to be biased, to think subjectively. 
In other words, we are incapable of being objective 
... try as we might. 

Consider the following sequence of numbers: 40- 

50- 60- . What is most likely the next number? 

70, of course. Buy why 70? There is an infinite num- 
ber of alternatives, some quite intriguing, as in 41- 

51- 61, 50-60-70, and so on. Yet, even though we 
may consider these alternatives, 70 will remain our 
preferred choice, because our minds instinctively, 
unconsciously perceive “40-50-60” as a pattern and 
are captured by it. And there’s absolutely nothing we 
can do to un-capture it. Why? Because that’s the 
way the human mind works. 

This simple exercise demonstrates that the mental 
machinery with which we think is inherently flawed: 
The Human Mind is Incapable of Being Objective. If 
the mind were really objective, it would not be capti- 
vated by the 40-50-60 sequence, and it certainly 
would not favor 70 as the next number over the lim- 
itless, more creative and more interesting alterna- 
tives. (Immanuel Kant, the great 18th Century 
philosopher, theorized that the mind is not designed 
to give us uninterpreted knowledge of the world, but 
must always approach it from a special point of view 
. . . with a certain bias.) 
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We are always prone to favor one side or another of 
an issue or problem because we interpret that issue or 
problem through the lens of biases and mindsets we 
acquire through our life’s experiences. The mind, 
unbidden and without our conscious awareness, cre- 
ates these biases and stores them away in memory 
where they serve as unconscious controllers of the 
myopic, custom-made mental lens through which we 
view and interpret the world around us. 

Our propensity to take sides — to think subjective- 
ly — is evident in the fact that we humans commonly 
“ begirt ” our analysis of a problem by formulating 
our “ conclusions We thus start at what should be 
the “end” of the analytic process. Therefore, our 
analysis of a problem usually focuses on the solution 
we intuitively favor. Accordingly, we pay inade- 
quate attention to alternative solutions; we look for 
and put store in evidence that supports our favored 
solution while eschewing evidence that does not, and 
at time we even maintain our support of the favored 
solution in the face of incontrovertible, contradicto- 
ry evidence. The human mind really is a piece of 
work! 

So what can we do about it? Or are we condemned to 
be ever victimized by our troublesome mental pro- 
clivities? 

There are two things we can do. First, we can quit 
thinking that we’re objective analysts. We are not. 
Humans are simply not objective. Second, we can 
organize — structure — our analysis in a way that 
ensures each element, each factor, of a problem is 
analyzed separately, systematically, and sufficiently. 

There are many different ways to structure analysis. 
My most recent book, The Thinker’s Toolkit: 
Fourteen Skills for Making Smarter Decisions in 
Business and in Life, describes some proven ones: 
problem restatement, pros-cons-and-fixes, sorting, 
chronologies, causal-flow diagramming, matrices, 
decision and probability trees, weighted ranking, 
hypothesis testing and utility analysis. All such tech- 
niques, by separating the elements of a problem in a 
logical, organized way, enable us to compare and 
weigh one element against another and to identify 
which factors and relationships are critical. Most 


importantly, these techniques compensate for the 
mind’s lack of objectivity by compelling us to sys- 
tematically consider alternative options and scenar- 
ios. Failure to consider alternatives is a principal 
cause of faulty analysis. 

Structuring is to analysis what a blueprint is to build- 
ing a house. Building a house, building anything, 
without a plan is, to say the least, ill advised. And 
what structuring is to a blueprint, the techniques of 
structuring are to a carpenter’s tools — not compo- 
nents of a unified system for analyzing problems but 
an assortment of techniques that can be used singly 
or in combination. 

Finally, structuring is not a substitute for thinking. It 
is rather a means to facilitate and empower thinking. 
Used properly and creatively, techniques for struc- 
turing will significantly enhance our ability to ana- 
lyze, understand, and solve problems, lead to more 
effective analysis and sounder decisions, and make 
us feel better about those decisions. 

Devil’s Advocacy 

One of the easiest structuring techniques — and a 
highly effective one — for countering our subjective 
tendencies is Devil’s Advocacy, which seeks to prove 
a contrary or opposite view to the one that is favored. 
The power of devil’s advocacy resides in our uncon- 
scious compulsion to favor an outcome or solution 
early in the analytic process. By artificially favor- 
ing — focusing on — a contrary or opposite view, 
devil’s advocacy activates our instinctive, subjective 
modes of thinking: paying insufficient attention to 
alternatives, looking for and putting store in evi- 
dence that supports the facile view and holding fast 
to the view in the face of contradictory evidence. 

Devil’s advocacy is thus indifferent to the favored 
view, and that is the technique’s principal strength — 
freeing the analyst to seek and obtain new evidence 
which was not sought in analyzing the favored view 
or, if obtained, was not believed. This thirst for, and 
receptivity to, evidence that contradicts the favored 
view is devil’s advocacy’s secret weapon, the extra 
dimension that makes it a formidable analytic tech- 
nique. 
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It’s easy to apply devil’s advocacy because we don’t 
have to learn any new analytic approach or device. 
We just follow our natural inclinations and let devil’s 
advocacy do the rest. But there is always strong 
resistance, both within the analyst and within a peo- 
pled organization, to taking, or even recommending, 
the devil’s advocacy approach. 

Imagine that you have just come up with a great idea 
for making your company rich, for which your career 
and pocketbook will benefit handsomely. How psy- 
chologically motivated are you to find and give cre- 
dence to evidence that your idea won’t work or that 
some other idea will make the company greater prof- 
its? Not much. Or imagine that a senior manager in 
the company has conceived of a promising new ven- 
ture and is pushing it. How receptive will that man- 
ager be to a proposal to gather and analyze evidence 
showing that the venture as originally conceived is 
flawed or that another venture offers greater 
promise? The very idea of undertaking a devil’s 
advocate approach is naturally interpreted as threat- 
ening to those who have endorsed the primary (the 
favored) view. 

Consider the hypothetical case of a large manufac- 
turing company which, despite aggressive advertis- 
ing, is faced with rapidly declining sales of its prin- 
cipal product. The company’s management has 
determined there are essentially two options: contin- 
ue production of the product with modifications to 
improve its appeal, or terminate production. If I 
were the company CEO, I would establish two com- 
peting working groups, one to seek evidence in sup- 
port of continuing production, the other to seek evi- 
dence in support of termination. I would charge each 
group with presenting their findings to the board of 
directors, which would then make the decision. To 
assign these two inherently conflicting analytic tasks 
to a single working group would be tantamount to 
letting a single lawyer both prosecute and defend 
someone in court. 

We can, of course, employ the devil’s advocate 
approach even when we are doing the analysis our- 
selves, alone. We simply work one view of the prob- 
lem and set our conclusions aside for a day or two to 


let our focus, mindset, and bias relax and fade a bit. 
We then go to work on the other side, trying to prove 
just the opposite with different evidence. 

Whether conducted by competing groups or a single 
individual, devil’s advocacy will, with virtual cer- 
tainty, open the mind of the analyst to new dimen- 
sions and perceptions of the problem, poking holes 
in fallacious, self-serving arguments and stripping 
away poorly reasoned and thinly supported evidence. 
That’s the wonder and delight of the devil’s advocate 
approach. 

Separating Utility and Probability 

Another troublesome feature of our minds is our ten- 
dency, when analyzing and discussing options for 
solving a problem, to address what we seek to gain 
from a particular course of action (that is, the utility 
we see in it) at the same time that we address the 
probability that this course of action will produce the 
desired outcome. Separating the analysis and dis- 
cussion of utility and probability is essential to 
objective analysis, because these are fundamentally 
different subjects, each with a different focus and, 
especially, a different language. Issues are raised 
and positions voiced in analyzing utility that are 
absent in analyzing probability, and vice versa. 

Utility Question: If we implement “Option A” 

and “Outcome X” occurs, 
what is the utility (the benefit, 
the advantage)? 

Probability Question: If we implement “Option A,” 

what is the probability 
“Outcome X” will occur? 

Listen, when colleagues discuss alternative courses 
of action. They will casually, unconsciously, switch 
back and forth between utility and probability, often 
in a single sentence, blissfully unaware they are 
doing so and unaware of the consequences. 

The district manager has convened a meeting of her 
sales staff. "Sales of our Super FAX 5000 are slip- 
ping, ” she declares. "What can we do about it?" 
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Jack: “Offer complimentary rolls of FAX 

paper. ” 

Manager: “Not a bad idea. That might interest 
some customers [Utility], but it probably 
wouldn 't last [Probability]. ” 

Jill: “How about offering extended mainte- 

nance warranties?" 

Manager: “I like that. The 5000 is very reliable, so 
it wouldn ’t cost us much [Utility]. ” 

Jill: “It might [Probability] even save us 

money [Utility]. ” 

When we mix elements of utility and probability 
together, we confuse the issues and muddy the ana- 
lytic waters because the assumptions, biases and pre- 
conceived notions that drive our assessment of utili- 
ty are entirely different from those that drive our 
assessment of probability. Our assessment of utility 
determines which option is most attractive. Our 
assessment of probability determines which outcome 
is most likely. In other words, utility determines 
what we want, probability what we get. 

To avoid the adverse consequences of intermingling 
these two basic components of analysis, I recom- 
mend addressing utility first, by asking the Utility 
Question of each option: If we implement the option, 
what benefit, profit, or advantage does its outcome 
provide? Then rank the options by the comparative 
utility of their outcomes. Spend some time at it. 
Ignore the probabilities for the moment. You’ll be 
amazed at how focusing your mind on just utilities 
empowers your thinking. When you are comfortable 
with your rankings, then and only then address the 
probability of these outcomes by asking the 
Probability Question. For example: 


Utility Rankings of 
Desired Outcome 

Probability of 
Desired Outcome 

Option C 

10% 

Option A 

50% 

Option E 

70% 

Option B 

40% 

Option D 

90% 


You will find that separating analysis of options into 
two steps is easy because it simplifies the process 
and, as I said, empowers the mind by enabling it to 
focus on one element at a time: first utility, then 
probability. 

But then what? How do we combine the utility rank- 
ings with the probabilities? We do it with an inge- 
nious device called Expected Value. We compute 
expected value by multiplying the utility of an out- 
come by its probability of occurring. This is easily 
done if utility can be expressed in terms of dollars. 
But if it can’t, we quantify utility on a scale of 0 to 
100, where zero is the least utility and 100 the most. 
We then multiply the utilities by their probabilities to 
determine their expected values. 


Utility Value of 
Desired Outcome 

Probability of 
Desired Outcome 

U x P = EV 

Option C 

90 

10% 

90 x . 1 =9 

Option A 

70 

50% 

70 X .5 = 35 

Option E 

30 

70% 

30 x .7 = 21 

Option B 

20 

40% 

20 X .4 = 8 

Option D 

10 

90% 

10 X .9 = 9 


In our example, Option A is strongly preferred. It is 
noteworthy that neither the option with the most ben- 
eficial outcome (Option C: 90) nor the one with the 
most likely outcome (Option D: 90%) emerged as the 
favorite. Option C had too little probability, and 
Option D had too little utility. By integrating utility 
and probability into a single quotient, Expected 
Value affords us a powerful and reliable means of 
evaluating, comparing and ranking options. 

The only way to learn devil’s advocacy, utility analy- 
sis or any other structuring technique is through 
practice. So try it. You’ll be surprised how structur- 
ing opens up the complexities of a problem and pro- 
duces valuable insights into its solution. Such is the 
power of structuring your analysis. 
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TechTracS 

TechTracS : NASA’s Commercial Technology 
Management System 

by Kevin Barquinero & Douglas Cannon 


In response to the Administration’s technology pol- 
icy, the National Performance Review and the 
needs of our nation’s industries, NASA 
Administrator Daniel Goldin issued NASA’s 
Commercial Technology: Agenda for Change in 
July 1994. The Agenda for Change outlines the 
national technology policy, Agency decisions made 
to implement the national policy, and the Agency’s 
newly defined Commercial Technology Mission. 
This paper explains the mission and TechTracS, the 
program’s commercial technology management 
system. 

Since its inception, NASA has recognized that the 
technology it develops in the course of its missions 
has relevance to the general economy. 
Consequently, the Agency has maintained a technol- 
ogy utilization program to transfer this technology to 
industry, but this early program had at best a passive 
relationship with industry. NASA disseminated its 
technical information routinely and reacted to com- 
pany inquiries as they came up. Serendipity was the 
“management system.” 

Today, economic development is an important 
national goal. Marketing NASA technologies, creat- 
ing new business practices for entering into partner- 
ships with industry, establishing and reporting met- 
rics, creating and updating an electronic network to 
better serve our customers, and implementing a 
training program to educate NASA employees to 
effect change in our culture are the main elements of 
the new, proactive commercial technology program. 
Taken together, these activities represent a funda- 
mental shift in how the Agency works with industry 
to commercialize its aeronautics and space technolo- 
gies. No longer is NASA relying on serendipity. 
Rather, the Agency is actively working to move our 
knowledge from our programs and laboratories 
through companies to the marketplace. 


The core process in this new way of doing business 
is “knowledge management.” NASA civil servants, 
contractors and grantees frequently create new 
knowledge of technology during its aeronautics and 
space missions that has commercial value embedded 
in it. First capturing and then managing this knowl- 
edge are the most critical functions of the commer- 
cial technology offices at each field Center. Without 
control of our technical knowledge we are handi- 
capped in our ability to maximize the number of 
NASA-industry collaborations. Conversely, having 
a complete database of all NASA technology invest- 
ments, along with an assessment of the commercial 
potential of these technologies, will greatly enhance 
the process of matching NASA technologies with 
industry needs. Spurred on by NASA’s 1995 
Strategic Plan calling for a 100% inventory of NASA 
technology for commercial potential, a small team 
set out to develop this knowledge management sys- 
tem. 

TechTracS Management Stages 

The purpose of TechTracS is to identify and capture 
all NASA technologies with commercial potential 
into an off-the-shelf database application, and then 
track their progress. As such it is, in essence, an 
“asset management system” much like those found 
in successful corporations. This management system 
consists of four stages: 

• The first is to develop an inventory of the 
Agency’s entire technology portfolio and 
assess it for relevance to the commercial mar- 
ketplace. NASA has already established an ini- 
tial operating inventory database and is one of 
the first agencies to do so. The commercial 
assessment is the responsibility of each NASA 
associate administrator and is conducted by the 
field center managing the technology activity. 
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• Those technologies that are identified as hav- 
ing commercial potential will then be actively 
marketed to appropriate industries. This is the 
second stage. Making our technology invento- 
ry available over the Internet is a key step in 
this stage. Such valuable information is thus 
delivered quickly and evenly to all who seek it. 

• The third stage is when a NASA-industry part- 
nership is entered into for the purposes of com- 
mercializing the technology. The sum totals of 
NASA’s contribution to these partnerships are 
tallied to show progress in the partnership 
requirements specified by the National 
Performance Review. 

• The final stage is to track the technology’s suc- 
cess or failure in the marketplace. While this 
system is initially aimed at leveraging NASA 
technologies into the marketplace, it can also 
be used to better leverage our technologies 
across NASA’s internal missions as well as 
technologies across national initiatives involv- 
ing multiple federal agencies. 

In addition to these stages, TechTracS will track a 
number of other management processes such as 
patent execution, license negotiation and TechBriefs 
abstract preparations in order to assure complete 
transfer of NASA technology. 

Assessment and Inventory 

Technically, TechTracS is a distributed network of 
relational databases located at each NASA field 
Center and Headquarters. It is a client/server archi- 
tecture that has user-friendly interfaces and is plat- 
form independent. It was developed for NASA by a 
small team at the Research Triangle Institute using 
ACI US’ 4th Dimension™ client/server relational 
database. It is a virtual office that enables coopera- 
tive data management and services such as metrics 
analysis, Internet services, automated documents and 
letters, ad hoc reports, on-line clients, email services 
and multimedia capabilities. 

The effectiveness of TechTracS is evident by 
NASA’s success in meeting its strategic goal of 


assessing 100 percent of its technologies for com- 
mercial potential. Working with the comptroller’s 
office and the procurement office, we successfully 
merged their respective databases, each Center’s 
technology database, and a newly developed partner- 
ship database into a single relational database in 
TechTracS. For the first time NASA’s entire FY 
1995 budget of $14 billion was correlated with its 
procurements, technologies and partnerships (which 
account for nearly 90% of the Agency’s budget). 

For any given year NASA manages over 10,000 con- 
tracts, grants and cooperative agreements ranging 
across over 25,000 program areas. When combined, 
these create a matrix with more than 50,000 areas of 
unique work tasks which are then allocated to 10 
field Centers and Headquarters. These 50,000 work 
areas represent an annual NASA investment of 
approximately $12 billion. This entire structure and 
its set of relationships are modeled in TechTracS. 

From July 1 to September 1, 1995, we assessed more 
than two-thirds of these 50,000 areas. In that time, 
2,700 new technologies emerged and approximately 
10 to 15 percent of these areas have been assessed as 
having commercial potential. More than $600 mil- 
lion or about 5 percent our annual investment in 
these work areas qualify as technology partnerships. 

This is the first time that a Federal agency has con- 
ducted such an extensive inventory of its programs 
and technologies for technology transfer. The initial 
results are impressive, but as we improve our report- 
ing system and when both NASA staff and the pub- 
lic become more knowledgeable of it, we believe we 
will increase the annual number of new technologies 
created by a factor of three over the next five years. 
We also believe the percentage of our programs and 
technologies with commercial potential will increase 
to 25 percent over the next five years. Finally, by 
1 999 we expect to increase the amount of resources 
we invest annually in partnerships from 5 to 20 per- 
cent. 

Partnerships and TYacking 

TechTracS offers benefits beyond its enhancement of 
internal commercial technology management. It 
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TechTracS 


makes possible customer services that heretofore 
were impossible to offer. First, and most important- 
ly, companies now have an easy-to-use, searchable 
database to locate NASA technologies that may 
solve their problems, wherever that knowledge may 
be — at a NASA lab, a contractor facility, or a univer- 
sity. Making the human connection between the 
knowledge owner and the knowledge seeker is the 
first order of business in technology transfer. 
TechTracS accelerates this process. 

The next step in technology commercialization is that 
a relationship must be established between NASA 
and the knowledge-seeking company. Once this rela- 
tionship is established, relevant information regard- 
ing the new partnership is stored in TechTracS. The 
National Performance Review expects 10 to 20 per- 
cent of NASA’s budget to be in R&D partnerships 
with industry. Because of TechTracS, the Agency is 
able to report accurately to the Administration its 
progress towards meeting this goal. 

While partnerships are a measure of the relevance of 
NASA technology to the U.S. economy, they do not in 
and of themselves contribute to the economic well- 
being of the nation. Companies must take the NASA 
knowledge they acquire and apply it in a new or 
improved commercial process, product or service. 
For NASA, success occurs when the company makes 
new capital investments, creates new jobs, and/or sells 
new and improved commodities in the marketplace. 

TechTracS is able to capture these “success stories.” 
With immediate access to this data, NASA will be 
able to demonstrate to the Congress and the American 
people the relevance of its investment in aeronautics 
and space for advancements in science, technology, 
and contributions to the United States economy. 

FY 1996 Goal: Training 

With the assessment of NASA’s entire investment 
base complete and ongoing, the next goal for the 
NASA team is to train individuals to take advantage 
of this system. Two strategies are being pursued: 

• In partnership with NASA’s executive training 
professionals, an internal training course is 


being developed for NASA civil servants, con- 
tractors and grantees. This course will be part 
in a series of NASA training opportunities that 
instruct NASA managers, scientists and engi- 
neers on the importance of the Commercial 
Technology Mission, mechanisms for entering 
into partnerships with industry, TechTracS ’s 
role in tying this all together, and how to use 
TechTracS ’s information system. 

• In partnership with the TechTracS industry 
team, a similar training course is being devel- 
oped for companies most likely to benefit from 
NASA’s technology transfer. Like its in-house 
counterpart, this course informs the partici- 
pants of NASA’s Commercial Technology 
Mission and partnership options. In addition, it 
will train these individuals on how to access 
the publicly available portions of TechTracS 
remotely so that they can seek information 
about NASA technologies on their own for 
their benefit or on behalf of a customer. 

Industry training is key to the commercial exploita- 
tion of this information. No single individual, team, 
organization or network of organizations has enough 
knowledge to maximize the transfer and commer- 
cialization of NASA technology throughout the U.S. 
economy. The economy is simply too big and too 
complex. However, many individuals, joint teams, 
multiple organizations and even networks of organi- 
zations can maximize the transfer and commercial- 
ization of NASA technology throughout the U.S. 
economy together. TechTracS training is the empow- 
ering tool. Upon completion of this course the atten- 
dees will receive a NASA certificate attesting that 
they understand NASA’s Commercial Technology 
Program and are skilled in using TechTracS to locate 
NASA commercial technology. 

The continuing evolution of NASA’s commercial 
technology management system can be a major fac- 
tor in such industrial advances and economic devel- 
opment. Success stories will hopefully become com- 
monplace. In analyzing those success stories, 
TechTracS should be able to illustrate the value and 
importance of placing the right technology knowl- 
edge into the right hands at the right time. 
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Are We Missing Something? 


Today’s Management Technique and Tools: 
Are We Missing Something? 

by Ernest M. Hahne 


For decades, bookstore shelves have been filled with 
all manner of business guidance and management 
philosophy. Today we can choose from hundreds of 
software programs that, taken together, claim to be 
the solution to any management problem in any man- 
ner of approach. Every month our mailbags are 
overloaded with offers for better training or more 
effective consultation and business reengineering 
support. 

With this much help, why do so many of us and our 
organizations continue to perform below par? Have 
the gurus of management science missed something 
that we all need to know? Is problem solution just 
“too hard,” given the complexity of modem business 
and government requirements? 

I do not believe it’s “too hard.” I do believe some- 
thing has been overlooked. This paper describes an 
approach that was used to uncover this missing link, 
formulate a solution approach and test solution 
validity against in-process program needs rather than 
in a rarified laboratory environment. 

Test/demonstration results indicate that we do not 
need to develop any new management principles. 
Rather, we need only to change our technique and 
some processes we use for application of existing, 
well-known principles. This rearrangement of tech- 
nique and process application does require some 
modification and addition to our management tool 
set. However, revolutionary change is not called for 
and may, in fact, be counterproductive. 

Identifying the Missing Link 

Several study reports concerning numerous program 
failures within NASA, the DoD and industry in gen- 
eral prompted a search in the late 1980s for a miss- 
ing management process link. 1 Based on the author’s 


personal experiences as a program and systems man- 
agement practitioner and consultant, an obvious 
question arises: Why do so many ventures that 
appeared sound at startup continue to report “sur- 
prising” indications of pending or actual failure? 
How can this be, given industry’s significant invest- 
ments in employee training, skills, hiring and acqui- 
sition of the “latest” in management information sys- 
tem (MIS) capability? What, specifically, goes 
wrong? 

A similar question was asked in the mid-1960s by a 
small government team tasked to improve the exist- 
ing program acquisition and management practices. 2 
This team (with the author as a participant) reviewed 
numerous programs such as the FB- 111, C5A and 
MinuteMan. We developed a lessons learned list of 
common reasons for major program problems. The 
list (unpublished at that time) was used as a guide for 
the creation of the MIL STD-499 Systems 
Engineering Management and early versions of the 
DoD 7000.2 Cost/Schedule Control Systems 
Criteria. The similarity between the data reported in 
the 1980s and in the 1960s list was very evident. 

A direct correlation yielded surprising results. The 
only difference between the two was the increased 
length of the 1980s list. 3 The 22 new items, resulting 
in a new total of 59 Failure Lessons Learned, related 
primarily to software development and integration, 
and the rest to funding issues. In the 1960s relative- 
ly few programs had significant software content, 
and funding was not the issue it is today. However, 
what was the explanation for the rest of the list? A 
sample of the expanded list is illustrated in Figure 2. 

Two approaches were addressed to explain the 
repeatability. The first, involving a validation review 
of existing techniques and processes, was rejected as 
time consuming and probably fruitless. Too many of 
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us have “been there, done that.” The approach taken 
was to search for a root cause, starting with the fun- 
damentals of the overall program/production and 
operations management process: specifically, how 
organizations convert input data and raw materials 
into products and services that are economically use- 
ful to an end user. Fundamentally, this was a repeat 
of a 1966 study (conducted by the author) that result- 
ed in a principle of management entitled “System 
Duality.” 4 


Program Failure Lessons Learned 

1 

Inadequate requirement specifications as part 
of the RFP severely compromise the overall 
acquisition effort and the quality of the 
delivered product. 

2 

Complete Interface Control Specifications 
between hardware and software and software 
are critical. 

3 

Adding manpower is rarely a solution to 
development schedule problem correction. 

4 

Training contractor and user personnel is 
essential. 

5 

Program management cannot specify good 
development criteria and just expect good 
development to happen. 

6 

Inadequately defined user requirements result 
in inadequate system/software specifications 
that lead to a contractually acceptable product 
that is operationally deficient. 

7 

Close and continuous monitoring at detailed 
schedule levels is essential. Risk Management 
needs should drive the level of detail. 

8 

Senior Management must be knowledgeable 
and involved in contract performance. 

9 

Communications and related documentation is 
critical to effective program configuration 
control and completion, i.e., ICWGs, minutes, 
telephone logs, Product Development 
Handbooks, etc. 

10 

Key personnel and management turnover 
causes critical problems. 


Figure 2. Program Failure Lessons Learned (Early 1990 
Compilation). 


The System Duality concept states that management 
always deals with two interrelated systems, as illus- 
trated by Figure 3. One is the organizational system 
(O) responsible for product production; the other is 
the product system (P) itself that is intended to satis- 
fy the end user needs. 

The concept also states that the key element of man- 
agement control over the process was the Transform 
Function, as illustrated by the overlap of the O and P 
systems. Thus, management control metrics would 
encompass planned versus actual cost, schedule and 
technical performance data, describing the (O) sys- 
tem conversion of inputs to deliver a product (P) to 
the user. 

The author’s re-evaluation of the concept supported 
its validity as described, and as applied by practice. 
Industry has reams of processes available to address 
all elements of Figure 3, with two exceptions. These 
are highlighted in Figure 3 by the items contained 
within the dashed boxes. These two items appear to 
be the missing link within our management process- 
es. Specifically, the absence of predictive and inte- 
grated risk analysis concerning the probability that 
our plans will fail at some significant cost and, also, 
our failure to assure timely review and feedback on 
developing results to the end user. In today’s com- 
mon practice, user feedback usually comes too late 
for easy design change. Essentially, the risks have 
already been incurred. 

Risk Management Planning 

Risk may be defined as the exposure to some likeli- 
hood of experiencing some loss. A loss can be 
expressed in many ways, such as a capability, eco- 
nomically, in terms of time, politically, socially, etc. 
The operative word in the definition is some. Loss 
magnitude can range from trivial to catastrophic. 
Loss occurrence can range from low to very high 
probability. Losses that do occur are usually addi- 
tive. 

There is always a likelihood of experiencing some 
loss. For previously demonstrated things, both the 
loss likelihood and magnitude may be known with 
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• Planned vs. Actual Data 



Risk that actuals will not 
equate to plan 


Figure 3. System Duality Concept. 

reasonable accuracy. For things not demonstrated, a 
significant range of uncertainty may exist concern- 
ing not only both parameters, but also the mecha- 
nism responsible for the exposure. 

Another finding from the Figure 3 study was that 
many causes for program failures appeared as the 
result of planning errors of omission. Items 1 and 6 
on the Figure 2 list exemplify this. The complete list 
provides several additional examples of planning 
inadequacy that suggest the need to change our basic 
planning concepts. 

First, we must admit that our biggest planning prob- 
lem is that we don’t know what we don’t know at 
process startup. The author and others call this the (I 
DON’T KNOW) 2 problem. If we don’t know that an 
issue exists, how can we possibly plan to avoid it? 

Fortunately, there are many tools available that, if 
used properly, would surface critical planning ques- 


tions. Unfortunately, too many of us do not use them 
or are unaware of their existence. 

One such tool is the list represented by Figure 2. Its 
use as a checklist is extremely valuable for risk 
avoidance planning. Several other similar tools will 
be described later. 

Another concept we should embrace involves the 
notion that in the absence of risk, management 
becomes basically unnecessary. Stated another way, 
we should conclude that the primary purpose of man- 
agement planning is to provide a roadmap and mea- 
surements for avoidance and/or control of risks that 
attend development of any new product. On aver- 
age, most of us currently practice reactive risk man- 
agement. We must change our practice to emphasize 
preplanned or predictive risk management. 

Another challenge to conventional thinking is that 
risk taking is bad. We can advance only by taking 
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risk. The key here is that any risk taken must be 
affordable. 

Finally, we must all realize that predictive risk plan- 
ning requires a greater investment of time, skills and 
experience. Numerous studies show that significant 
payback can result when an upfront investment is 
dedicated to more detailed planning. Figure 4 illus- 
trates data from two such studies. 

The above recommended changes to our conceptual 
planning approaches are illustrated pictorially in 
Figure 5. Our current approach is illustrated at the 
top left. We have a plan for concept A with no up- 
front risk assessment. Implementation results are 
illustrated at the right. Note that “surprise” risk loss- 
es are a significant part of total cost, that total cost 


exceeds what was planned, and that a part of planned 
value was lost due to risks having occurred. 

Just below we show the same Concept A plan but 
have included risk assessment. Note that total cost 
now includes the risk cost. Of course, the advertised 
cost is higher than one that did not include risk costs. 
Would the second plan and price be a winner? 

An alternative plan (Concept B) including its risk 
costs is shown at the bottom of Figure 5. Note that 
total cost as illustrated is physically smaller than A 
above it and also that the risk budget is smaller. 
Planned value results remain approximately the same. 
(B results from trade studies that improve the baseline 
of A.) This figure illustrates the objectives that man- 
agement techniques and tools are intended to achieve. 
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Figure 5. Risk Planning Concept. 


Systems Engineering: A Primary Risk Analysis 
Technique 

Risk analysis has a clearly definable starting point. 
Specifically, that point includes complete and quan- 
tified definition of end user needs, related constraints 
and measures of effective user results. This is an 
iterative process. The existing classical processes 
for systems engineering provide the foundation for 


performing predictive risk analysis and planning. 
This process starts with the end user needs and con- 
cludes with the assured delivery of an acceptable end 
product. 

This paper does not address systems engineering 
process applications for resolution of all risk analy- 
sis needs. The applications that are addressed focus 
on how risks within a design concept are surfaced 
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and how relative measurements can be made con- 
cerning their probability of occurring and the magni- 
tude of loss if they occur. These relative measure- 
ments will serve as a flag and guide to management 
for their investment of resources and attention to 
avoid or control each identified risk. 

An overview of the systems engineering process as 
commonly discussed in most publications 5 is shown 
in Figure 6. Note that risk analysis is one of many 
supporting functions to the centralized functions of 
system evaluation, trade studies and optimization. 


Conclusions concerning the repeatability of the 
Figure 2 Program Failure Lessons Learned suggest 
that Figure 6 should be revised as shown in Figure 7. 
These revisions should aid future system engineering 
practice as needed to achieve predictive risk plan- 
ning and more certain risk control. All suggested 
revisions can be correlated to one or more Program 
Failure Lessons Learned. 

Revision 1: Insert risk analysis within the central- 
ized function block. As a supporting function, many 
interpreted it to be a standalone requirement. 



Figure 6. The Classical System Engineering Process. 



Figure 7 . Suggested Revisions to the System Engineering Process. 
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Factually, that is how it is treated throughout today’s 
DoD 5000 Series, NHB-7120, and most other publi- 
cations. The transform function, previously shown 
in Figure 3, requires that risk analysis must be inte- 
grated within and across all functions. 

Revision 2: Add the end user as a major function at 
the process beginning. Most of us overlook the crit- 
icality of this function to systems engineering suc- 
cess. Initial user inputs should only be introduced to 
block 1, Mission Analysis. Future inputs should be 
introduced into both block 1 and block 5, System 
Definition. Feedback should only emanate from 
block 1 1 . 

Revision 3: Clarify that all communications between 
the centralized and supporting functions are two-way, 
and for new problems, real time. Use double ended 
arrows. These paths contain the data for process 
direction, authorization and reporting of process 
problems and results. Real time communication con- 
trol is critical to effective conduct of the Successive 
Refinement Process of Systems Engineering. (Avoid 
surprises at major progress reviews.) 

Revision 4: Annotate block 5, System Definition, to 
emphasize that the purpose of the entire process is to 
select a best alternative based on a trade study among 
alternatives. Too many programs fail because the 
trade study was inadequate or not conducted. 

Revision 5: Add Configuration Management (CM) 
as an administrative support process within systems 
engineering. CM should not function as a decision 
authority for change or approval. Reserve this role 
for the centralized authority of block 11. Also, all 
trade study data should be controlled under CM. 
Trade study results and decisions are totally depen- 
dent on the assumptions made and the analytical 
technique used. If these data are not available for 
future change analysis, chaos can result. 

Trade Study and Risk Planning 

Effective risk management depends on trade study 
performance and trade study is the heart of systems 
engineering. Systems engineering and risk manage- 
ment are totally intertwined. 


While the actual performance of a trade study is usu- 
ally complex and difficult, the fundamental concept 
is easy. (See Figure 8.) 



Quit 


Figure 8. A Simple Trade Study Process. 

If a first pass through steps 1 to 4 don’t yield a yes (it 
usually won’t), exercise paths 5, 6 and 7 singly or in 
parallel. At this point block 4 becomes the trade 
study function where the best of all available choic- 
es is tested for acceptability. If a yes is not obtained, 
repeat 5, 6 and 7 or decide you have no acceptable 
solution approach and go to path 8 “Quit” or No Bid. 

Obviously, a first step is to define an initial candidate 
solution that demonstrates feasibility for satisfaction 
of end user needs. Since this paper is primarily 
about techniques that avoid or mitigate risk, three 
major recommendations must be made concerning 
step one. First, obtain every scrap of detail available 
concerning user needs, related constraints and mea- 
sures of minimally acceptable performance of the 
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product/system to be delivered. Help the user create 
this data if it is inadequate. Second, make sure that 
only highly skilled engineers experienced in the dis- 
ciplines needed for initial solution definition are 
assigned. Third, avoid elegance in first cut 
approaches. Emphasize substance of need and why 
off-the-shelf solutions may be inadequate for user 
need satisfaction. Failure to adhere to the above rec- 
ommendations will increase startup cost and may 
result in unforeseen life cycle risk in resulting pro- 
gram plans. 


Design Synthesis 

Creating the initial system solution candidate 
requires most of the functions of the System 
Engineering process illustrated by Figure 7. Initially, 
blocks 1 , 2, 3, 4 and 6 are most critical. Difficulty in 
creating their data products suggests that team expe- 
rience may be inadequate or that for block 4 the 
existing technological art is too limited. The latter 
issue represents a major risk that is discussed later. 
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Without belaboring how the Figure 7 processes are 
performed, the synthesized system concept that 
results from block 4 could result in a model as shown 
in Figure 9. At the top left are stated user needs 
(ra to rc) that initiate the analysis process and defin- 
ition of specific system functional requirements, fj 
through f 5 . These functions are allocated to subsys- 
tem B], B 2 and B 3 . Further functional decomposi- 
tion occurs and, as shown for B 2 , these subsystem 
functions are allocated to end items Cj, C 2 and C 3 . 
They provide the capabilities to perform the system 
and subsystem functions: for example, Ca and Cb for 
end item Cj. 

A basic feasibility test of this synthesized design is 
conducted by asking the following questions: 

1. Can end item capabilities, as identified, be rea- 
sonably satisfied by existing or new equipment 
known to be undergoing development? 

2. Are there obvious reasons why the end items 
within the model would be difficult to produce or 
support logistically? 

3. Are there difficult and perhaps unacceptable 
engineering specialty issues related to reliability, 
maintainability, human factors, safety, etc., con- 
cerning any of the end items or their integration? 

4. Are the end items high cost? Is the schedule for 
their availability reasonable? 

A negative response to one or more of the questions 
requires repetition of the Figure 7 process until two 
or more alternative synthesis models that demon- 
strate feasibility of satisfaction of end user needs are 
defined. 

Note: If at least one feasible candidate cannot be 
defined, stop work. If this is due to unavailable tech- 
nology, consider initiating an R&D project. 

Establishing plausibility of each feasible design fol- 
lows the Figure 7 process but emphasizes efforts 
through blocks 7, 8, 9 and 11. These activities are 
complex, time consuming and relatively expensive. 
The Program Failure Lessons Learned List items 


(Figure 2) suggest they are among the most poorly 
performed systems engineering activities. However, 
without some reasonable data input from them, 
effective performance of the block 1 1 trade study is 
hopeless. 

Experience has shown that designing for perfection 
is infinitely costly and time consuming. Also, given 
the rapid growth of technology while we are design- 
ing, it’s probably impossible. We need to change our 
selection and approval paradigm from a search for 
what’s best, to a search for what is “least bad” but 
acceptable for satisfaction of known needs. 

I do not suggest eliminating classical system effec- 
tiveness and life cycle cost analysis processes. I do 
advocate doing them only in areas where user need 
satisfaction would be significantly impaired by their 
absence. For any other purpose they tend to waste 
resources and time. 

The following sections present a “poor person’s 
approach” to resolution of these measurement needs. 

Risk Management Decision Making 

The proposed poor person’s approach emphasizes 
the drawing of management decision attention to 
what most of us call grey areas. 

Critical issues are usually obvious early on. (They 
can be enhanced by the judicious use of past lessons 
learned checklists.) Once known, they are sometimes 
given more attention than deserved. 

Small issues are often set aside, as they should be, 
unless their impacts can be shown to grow. 

The vast majority of issues are somewhat vague and, 
unless prioritized relative to their potential contribu- 
tion to end user need and risk, consume vast amounts 
of management time and “self-protection” funding. 

In addition to prioritization, another concept drives 
implementation of the poor person’s approach. 
Rigorous mathematical analysis is often no better 
than relative magnitude estimation by an expert. 
Management decision making requires a “go/no-go” 
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approach to metrics, not the precision that results 
from sophisticated and often computerized methods. 
The latter are usually costly and add little extra 
value. 

Following is an application example of the poor per- 
son’s approach to decision making, using an example 
solution candidate matrix based on the simple model 
shown previously in Figure 9. This measurement 
application example is shown in Figure 10. 


Measurement begins at the upper left with user defi- 
nition of value for each stated requirement. (For this 
discussion, limit this to value statements concerning 
mission functional requirements shown as ra, rb and 
rc.) Measurement ends at the lower right of the fig- 
ure. This is where the engineer ranks the ability of 
available or soon to be available end items (hardware 
or software) proposed to provide the capabilities 
required to support satisfaction of mission function- 
al requirements. In between are subsystem alloca- 
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Figure 10. A Typical Candidate System Design Matrix with Value Measurements Annotated. 
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tion concepts which serve as design control mecha- Repeating the process for subsystem decomposition 

nisms. They allocate superior needs downward, to and allocation the end items making up B 2 have the 

end items intended to serve these needs. Thus, the following values: 
sum of user needs must be satisfied by the sum of 

end item capabilities. Management is concerned C] = 28 where Qa = 14 
with the risk that this equation may not be met unless Cib = 14 

they exercise decisions to assure they will be. C 2 = 16 where C 2 a = 14 
Simple step function metrics can effectively point C 2 b = 2 

the way. C 3 = 2 where C 3 a = 2 


As shown in the Figure 10 example, the user states a 
value rating for each defined need, using a scale of 
10 for highest value and 1 for lowest value. 
Intermediate values fall in between. In the example 
shown requirement (ra) is valued at 4, (rb) at 8 and 
(rc) at 2. 

Based on systems analysis, the engineer has identi- 
fied five major functions (f j through f 5 ) as needed to 
satisfy the user requirement. How these functions 
contribute to user requirement satisfaction are shown 
by the dots at intersections of the (ra) (rb) and (rc) 
lines with the vertical function lines. 

To assign values to functions, each dot is given the 
value of its source requirement. To establish a 
functions value, add up its vertical dot values. 
Thus: 

ft = 6 
f 2 = 8 
f 3 = 4 
f 4 = 2 
f 5 = 14 

One reason that f 5 is so high could be that its design 
represents a centralized computation function that 
contributes to performance of all other functions. 


At this point a relative value for all synthesized capa- 
bilities of a given design concept are established. All 
originate from stated user needs and values. Notice 
that the arithmetic method used amplifies the value 
numerics that flow downwards from the user mission 
requirements. Based on these value assignments, 
management attention should emphasize end item Cj 
of B 2 over C 2 of B 2 . However, until the risk associ- 
ated with the acquisition and delivered performance 
of each end item is understood, management atten- 
tion based on value alone may be misdirected. 

While system value analysis is performed “Top 
Down,” system risk analysis is performed from the 
bottom up. Consider the following axioms. 

Axiom 1: Functional and physical performance of 
systems and subsystems is only limited 
by the capabilities of their end items. 

Axiom 2: Systems and subsystems don’t fail. Only 
their end items do. 

Axiom 3: End item risk is a function of its maturity 
and past performance history. If an end 
item’s capability has not been demon- 
strated previously within its intended 
operating environment, it is risky. 


Subsystems of the synthesized design are shown as Axiom 4: Planning granularity is the most critical 
B],B 2 andB 3 . Each is allocated the subrequirement requirement for early surfacing and 

to perform all or part of the system functions f] assessment of risk. End items must be 

through f 5 . Again, allocated functional values are understood, 

added and the relative subsystem valuer rcome: 

Given the above, the author suggests the use of data 
Bj= 12 as shown in Figure 1 1 as a tool for assigning a Risk 

B 2 = 16 Index to the capabilities of end items as synthesized 

B 3 = 6 for a new system. Note that the highest end item risk 
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Risk Index* 

Risk Characteristic 

10 

New Technology Required 

7-10 

New development: Technology exists, 
but unproven for this use 

5-8 

New Design: Similar equipment in use. 
None directly applicable to this need. 

3-6 

Design Upgrade: Similar equipment in 
use: > 40% change required. 

2-4 

Shelf Modification: < 40% change 
required. 

1-2 

Shelf Equipment: COTS: Only changes 
as required for integration. 

* Note : The risk represents your resources expendi- 
ture to achieve the user requirement. The more you 
must invest, the greater your risk of loss. 


Figure 11. Risk Index. 

End Item Maturity/Characteristics vs. Risk. 


characteristic is assigned a 10 while the lowest is 
assigned a 1 or 2. A Risk Index equal to zero is never 
used. End items assigned a value of 8 or higher 
should be considered a candidate for R&D or Pre- 
Planned Productivity Improvements (P3I). 

Apply the Risk Index data of Figure 1 1 to the exam- 
pled synthesis in Figure 10. Sample results are 
shown in Figure 12, and are explained as follows. 

1. The value of a capability (V) multiplied by its 
Risk Index (RI) equals the Management 
Concentration Index (MCI). Management 
should focus on capabilities that have highest 
value and risk combinations, i.e., V x RI = MCI. 

2. Based on technology status, a Risk Index (RI) is 
assigned to each capability. (See lower right of 
figure.) 

3. The capability value assignment (V) and (RI) are 
multiplied to obtain each end item capability 
(V x RI). 


4. Add the capability (V x RI) totals to obtain the 
end item (V x RI). 

5. Add the end items (V x RI) to obtain the subsys- 
tem (V x RI). 

The resultant data per Figure 12 could be normalized 
to suggest that management attention for allocation 
and control of resources for Subsystem B 2 be applied 
as follows: C, = 43%; C 2 = 54%; C 3 = 3%. 

The same processes could be applied to Subsystems 
B 1 and B3 end items. Normalizing all data across 
subsystems would result in relative ranking of all end 
items to prioritize management concentration across 
subsystems. 

In a similar manner, subsystems could be ranked. By 
continuing the flow upwards to the system level, a 
system (V x RI) or MCI metric results. Given that, 
alternative syntheses can be compared to determine 
which one has a best change of being “least bad.” 
Also, the detailed metrics data provides an indication 
of the plausibility of continuing with efforts for 
detailed design of the least bad alternative. 

The reader should understand that the above numer- 
ics have only addressed technical needs risk assess- 
ment. The process can be expanded to encompass 
both cost and schedule parametrics as necessary to 
support more robust management decision making 
guidance. Economic rather than engineering deci- 
sion theory provides the basis to such expanded 
application. 

Supporting Tools and Training 

On average, no new tools are required to perform 
what has been described. Most of the arithmetic 
processes presented can be aided by basic spread- 
sheets and a simple relational database. 

Extending the technical risk assessment process to 
encompass economic issues requires tools and tech- 
niques that are generally unfamiliar to most systems 
engineers. 
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Figure 12. A Typical Candidate System Design Matrix with Value and Risk Measurements. 


While new tool requirements are not a major issue, 
the failure or inability by most of us to use existing 
tools properly is a major issue. Some examples: 

Checklists: Dozens exist in the literature that are 
rarely used. Using them can reduce risk that derives 
from errors of omission. They will jog the experi- 
enced person’s memory. For inexperienced people, 


they stimulate questions and thought. Once an issue 
is surfaced, resolution will be addressed. Most 
checklists have been developed because of recurring 
failures. 

Specification Formats: When combined with their 
descriptive instructions, they are a checklist. Don’t 
modify their content. Tailor your response detail. 
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Mark unapplicable items as N/A. That is a useful 
data element to your reviewer. 

Data Requirement Lists: Same value as the above, 
with one additional thought. If an item of data is 
necessary for decision making and future 
product/system maintenance or change, produce it. 
All else should be avoided. 

Software Systems: Don’t buy the latest because it’s 
there. The cost of training and equipment upgrade 
can be prohibitive. Stay with what is “least bad,” 
that with lowest risk. 

Training is or should be a major concern, but most 
organizations continue to regard training from view- 
points that do not and cannot satisfy today’s business 
and program management needs. Some specific 
issues of concern are: 

Formal Training: Too many organizations continue 
to provide training from a “Square Filing” view- 
point. A person must participate in so many class- 
room hours per year to be considered for advance- 
ment. As an alternative, we should be training peo- 
ple to help them make decisions about things they 
are accountable for. Can it be that we don’t know 
what their accountabilities are or should be? We 
should test every student in terms of how job perfor- 
mance was improved (risk reduction) because of 
classroom attendance. 

Training Curriculum: Most training continues to 
teach the basics. While important, these are not suf- 
ficient. In today’s business environment training 
must be tailored to fit the student’s working needs. 
Basic theory, coupled with a generic classroom exer- 
cise, is usually too vague for timely job application 
subsequent to course completion. Solution of this 
problem involves two considerations. First, empha- 
size training of an Integrated Product Team (IPT) 
rather than a general student group. Secondly, tailor 
all training and classroom exercise to definition and 
management of the IPT’s joint responsibilities and 
accountabilities. 


Basic IPT training should emphasize teaching the 
overall processes of Program and Systems 
Management as required to meet IPT needs. This 
basic training should be followed up with specialty 
courses for the team after unique needs are deter- 
mined as part of on-the-job training (OJT). 

On-the-job Training: Tailored formal training with- 
out the provision of OJT has been shown to be inef- 
fective. The classroom exercise should be developed 
as the OJT start-up exercise. Essentially it should be 
the “plan for the plan” of the IPT to develop an inte- 
grated IPT Project Plan after formal training. This 
planning effort identifies the need for follow-on spe- 
cialty training courses. The earlier discussions of 
this paper outline a “plan for the plan” approach, 
resulting in a capability for risk management deci- 
sion making. 

Mentor Support: All but absent in most organiza- 
tions today, mentor support is proving to be a costly 
issue for many organizations. It represents a form of 
training that is impossible to formalize for two rea- 
sons. When it’s needed, it’s needed now. And, what 
is needed can only be derived from combining previ- 
ous experiences. There are two approaches to serve 
this need: retain some top quality “oldtimers” for this 
purpose, or, be sure that the selected IPT/OJT 
instructors can provide the service. A little of both 
may be the best choice. Consultants are not usually 
effective in this role. 

Industry Lessons Learned 

Over the past two years, the processes described in 
this paper have been applied to several NASA, DoD 
and commercial projects. In each case, formal train- 
ing, OJT and mentor support was provided to an IPT. 
Descriptive experience concerning each project’s 
results are beyond the scope of this paper. 6 However, 
the following lessons learned are typical of each. 

1 . A young team can follow the requirements of the 
NMI-7120 and DoD 5000 series processes with 
adequate training, OJT and mentor support. 
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2. You can start process application in the middle of 
a project. 

3. Positive results are achieved within six to 12 
weeks of start-up; that is, by a next-scheduled 
review. 

4. At the next-scheduled review, there is more 
information on the scope of the effort and poten- 
tial risk identification than by following the 
“usual” process, for the same or less effort cost. 

5. Processes can force identification of risk areas 
that need to be addressed early on. 

6. Specification/product trees can define an analyt- 
ical baseline for planning, even if initially incor- 
rect. 

7. Help is essential in determining appropriate 
process tailoring. 

8. The process holds people accountable and relies 
on hard data and metrics to determine perfor- 
mance acceptability. 

9. The process provides high visibility over issues 
that affect interfacing projects. 

10. The Planning/War Room process provides an 
effective means for evaluators and management 
to review work in process rather than waiting for 
a scheduled review. Reviews are shorter and 
fewer discrepancies are noted. 

1 1 . Planning/War Room data appears more complex 
and labor intensive than the usual process. It’s 
not! 

12. Resource-Loaded Schedule and Life Cycle 
Costing is not hard. It forces one to think about 
what is being done versus what should be done, 
and it surfaces uncertainty for early risk planning. 

13. System/concurrent engineering is critical. End 
users must be involved at start-up. 


14. In the beginning, some false starts will be made, 
but that is part of the learning process. 

15. Management must provide proactive support to 
process implementation. 

Failure Lessons Learned 

Comparison of the Failure Lessons Learned in the 
1980s with those from the 1960s showed them to be 
basically the same. I concluded that something was 
missing in how we were performing management. 

A new analysis of the very basic requirements under- 
lying the management process revealed that little if 
any emphasis was given to the management of risk. 
In general, it was observed that risk management 
was conducted to fill a square. Risks were only 
treated seriously when they had already been 
incurred. Few if any programs addressed predictive 
risk management. 

A subsequent analysis of the Failure Lessons 
Learned List in the light of predictive risk manage- 
ment objectives revealed that some modest changes 
to existing practice could yield significant return. 
Following are some specific changes that have been 
presented. 

1. Risk must be taken in order to advance or 
improve. The purpose of management is to sur- 
face and avoid unacceptable risk. 

2. Early and in-depth planning is the only tool that 
can surface risk and thereby avoid reactive risk 
management. You must plan to a level of granu- 
larity that assures all remaining risk is affordable. 

3. If remaining risk is not affordable, but the goal is 
valuable, consider an R&D or P 3 I program in 
place of a Development/Production Program. 

4. Management must redefine their decision crite- 
ria to choose the alternative that is “least bad” 
yet still meets overall end user system require- 
ments. 
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5. Systems engineering must be recognized as the 
primary discipline that provides a common 
thread among all program management disci- 
plines. 

6. Simple mathematical processes can serve to sup- 
port most value/risk decisions involved in the 
trade study analyses. 

7. Process application rigor is essential to perfor- 
mance of predictive risk management. 

8. Front end planning should be assigned only to 
experienced and skilled personnel. 

9. All personnel should be trained to understand 
their role in the systems engineering “Big 
Picture.” Such training provides the foundation 
to the Integrated Process Team’s performance as 
required to carry out the Systems Engineering 
process. The use of checklists should be a major 
training thrust. 

The results of the two studies have been presented 
for independent assessments of why so many major 
government programs are behind schedule, over 
budget and often deliver products that fall short of 
required operational capability. The studies were 
conducted more than 20 years apart, yet the failure 
reasons were basically the same. 

Based on early study results, many changes were 
made to existing management policy, practice and 
procedures. Based on the more current study, simi- 
lar changes are being made. 

Comparative review of these new requirements ver- 
sus the old revealed that the new practices are more 
clear and streamlined, but that no substantive differ- 
ences are evident. Thus it appeared questionable that 
the next 10 or 20 years would produce any more 
improvements than the last 20 years. Better training 
did not appear to be the answer per se. Since 1970, 
industry and the government have invested heavily 
for this purpose. I felt something was still missing 
from our approach. 


A return to basic analysis of fundamental business 
practice suggested this to be true. It was established 
that the primary need for management was to avoid 
risk in the Program Development and Acquisition 
process. A review of old and new practice through 
NMI 7120, the DoD 5000 series and other similar 
policies, showed that risk was addressed poorly, if at 
all. 

This paper described a relatively simple approach 
towards solution of the risk management problem. 
The process is founded on the practices of our cur- 
rent systems engineering processes. Field testing has 
shown that predictive risk management is practical 
and not too hard to perform by a young team, given 
some simple checklist tools and minimal training in 
their use. 
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William E. Lilly, Project Director 


The National Aeronautics and Space Administration 
(NASA) has successfully managed some of this 
country’s most complex technology and develop- 
ment programs. These successes have included the 
application of sound program control processes. The 
impetus for this study arose from the NASA 
Management Study Group findings that, over time, 
some program control tools and disciplined proce- 
dures and processes had weakened. The Study 
Group recommended that steps be taken to establish 
a comprehensive training approach in program man- 
agement, and, specifically, in program control func- 
tions. This study looks at program control processes 
within NASA currently in use, defines a “model” of 
program control functions, and provides recommen- 
dations on program control training needs and 
opportunities. 

In 1988, NASA Headquarters tasked the National 
Academy of Public Administration (NAPA) to exam- 
ine the processes and systems used to by NASA to 
manage and control program and project activities. 
Essential elements of a program control system 
include program development planning and docu- 
menting program requirements; integrated schedul- 
ing; resources management; configuration manage- 
ment; documentation and data management; estab- 
lishment of essential baselines; and the conduct of 
performance reviews. Specifically the NAPA study 
was designed to include: 

• Determination and definition of program con- 
trol functions as currently practiced in NASA. 

• Definition of a model of program control func- 
tions for NASA. 

• Observations on training of personnel. 


• Generation of recommendations for training in 
program control objectives and processes at the 
basic, intermediate and advanced levels of pro- 
ject management. 

The impetus for a program control aspect of program 
and project management training and developmental 
efforts can be traced to a series of findings and rec- 
ommendations on strengthening program manage- 
ment and control functions, which were derived from 
the Rogers Commission and the NASA Management 
Study Group (the Phillips Committee) reports. In 
reviewing the total function of NASA program man- 
agement, the Phillips Committee found the weakest 
area to be that of program planning and control. 
Committee members commented that over time 
NASA’s use of program control tools and disciplined 
procedures and processes had weakened. They rec- 
ommended the reinstitution of a Program Approval 
Document system and a revitalized hierarchy of pro- 
gram/project status reviews against approved base- 
lines. In addition, the Study Group recommended 
that steps be taken to develop a comprehensive train- 
ing approach in program management, specifically 
in program control functions, that would be based on 
real experience. 

The significance of the program control functions 
within NASA cannot be overstated. The success of 
large and complex research and development pro- 
jects depends on commitment, diligent and disci- 
plined attention to numerous planning, resource and 
scheduling variables, and the integration and balanc- 
ing of complex, interrelated activities. Along with 
the systems engineering function, the program con- 
trol function is one of the most important activities in 
successful program/project management. 
Systematic and disciplined attention to the implica- 


33 


tions of variances between planned baselines and 
actual performance on development projects is criti- 
cal to taking early remedial action, reducing costly 
delays and achieving success. 

The purpose of this study is to indicate the areas of 
need as well as provide guidance to the development 
of training opportunities concerned with program 
control in support of effective program/project man- 
agement in NASA. The study could not have been 
completed without the assistance of NASA employ- 
ees at field Centers and Headquarters. Their contri- 
butions helped the staff to understand the application 
of project control functions at different Centers. 
Special thanks are owed to Frank Hoban, Program 
Manager, NASA Program and Project Management 
Initiative, who provided the Academy with the envi- 
ronment to pursue the study. 

The Program Control Function 

In NASA, a project . . is a defined, time-limited 
activity with clearly established objectives and 
boundary conditions executed to gain knowledge, 
create a capability, or provide a service.” Major 
space research and development projects in NASA 
typically include design, development, fabrication, 
test, and flight operations. A program/project man- 
ager is designated responsible for ensuring the per- 
formance of all functions necessary for management 
of the project. The three basic elements of the man- 
ager’s job are technical performance, cost and sched- 
ule. The program/project manager needs to know 
where the project is at any point in time and to iden- 
tify and scope problems early. Program/project con- 
trol, which aids the project manager in this regard, is 
the total management process of establishing and 
maintaining program baselines and effectively sup- 
porting the project manager in meeting the overall 
objectives of the project. 

The combination of functions of program control is 
an essential element of the program management 
process. The establishment of comprehensive per- 
formance requirements by systems engineering pro- 
vides the details and parameters necessary for pro- 
gram control to maintain a comprehensive, ade- 
quately explicit and integrated program plan. This 


plan documents and defines program requirements 
and establishes the official baselines of program con- 
tent, scope, configuration, schedule and cost. A com- 
prehensive program control process includes proce- 
dures for reporting and reviewing performance 
against baselines; analyzing and synthesizing pro- 
gram performance; evaluating alternatives; develop- 
ing disciplined processes for considering, approving, 
and implementing changes to official baselines; and 
assuring positive feedback on all directions and deci- 
sions. It also provides a uniform system of program 
documentation and assures clear and consistent com- 
munications throughout the program community on 
program progress, status and issues. The integrated 
operation of these functions furnishes the means to 
determine the harmony of actual and planned cost, 
schedule and performance goals during development 
and fabrication by verifying whether everything is 
occurring in accord with baseline plans. The larger 
point is clear: a program control system requires sus- 
tained attention to the system as a totality, rather than 
as a group of parts. 

Ultimate responsibility for the effectiveness of pro- 
gram management control rests with top manage- 
ment. Top management decides upon Agency strat- 
egy, policy, and organizational and accountability 
structure. The control system is a set of major tools 
and procedures for implementing those decisions 
and for forming coherent and defensible strategies to 
cope with changed and changing circumstances. For 
the most effective program management and control 
to exist, an environment of accountability of organi- 
zations and individuals needs to exist at the top of the 
Agency. It should be clear to the entire Agency how 
NASA intends to operate and what is expected of all 
elements. Delegations of authority, definitions of 
roles and assignments of responsibilities should 
carry with them the terms of accountability. 
Disciplined processes for obtaining required feed- 
back on delegations and for measuring and system- 
atically reviewing performance on programs and 
projects should exist. The pattern of program 
reviews against approved program baselines should 
also be established at the top. This can consist of 
separate reviews or be a part of the general manage- 
ment review process, but a disciplined approach of 
reviewing status against approved baselines by the 
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Administrator and/or the Deputy is needed. The 
strength of such an approach is that it allows Agency 
leaders to directly, programmatically and effectively 
keep tabs on the performance and potential pitfalls of 
programs. This in turn enables top managers to iden- 
tify and consider the implications of both “inside” 
factors and “outside” factors, forces and trends 
which are likely to have an effect on NASA and its 
missions. 

A number of characteristics distinguish NASA 
research and development projects, including: 

• Uncertainty. Many of the processes and prod- 
ucts to be developed will be undertaken for the 
first time and all components require the per- 
formance of advanced technologies. 

• Long lead times in development and fabrica- 
tion. This necessitates concurrent development 
of elements and subsystems and the fitting of 
end products together. It requires a high order 
of advance planning and detailed monitoring 
and tracking, and increases the need for testing 
(component testing, subsystem testing and sys- 
tem testing). 

• Size and complexity of projects and the large 
number and dispersion of participants. 

• Persistent scrutiny of projects by the public, the 
Congress and the scientific community. Not 
only must the work be done well, the project 
manager must be prepared to interpret, explain 
and defend what is being done and why. 
Practices and standards for public projects far 
exceed typical industry standards. 

Against this background, it is important to keep in 
mind that good program management is a matter of 
balancing different internal and external factors so 
that performance is maximized over the longer term. 
Program control interventions, if used correctly, help 
to maintain this balance. 


Major Functions of Program/Project Control 

The basic control functions for development projects 
are planning, configuration management, schedul- 
ing, resource management and data management. In 
some cases, procurement activities and other busi- 
ness management activities may become part of the 
control function, as well as logistics and separate 
activities for program analysis, management infor- 
mation and program reviews. The combination of 
activities included depends upon the size and com- 
plexity of the program or project, the existing sup- 
port structure and the preferences of the Centers and 
the individual managers. Regardless of the individ- 
ual functions, more than anything else in program 
control, it is important for the personnel to see and 
comprehend the totality of the job to be done and to 
thoroughly understand the interrelationships and 
interfaces of the subsystems and systems, as well as 
the organizations and participants in the project. 
Another important element in structuring and carry- 
ing out program control functions is uncertainty and 
the inability to completely eliminate it. Uncertainty 
should be specifically considered in program plan- 
ning, scheduling and resource planning. 

Program Plans and Requirements 

The development plan is the basic plan for execution 
of the program or project. It is the top-level require- 
ments document and the top-level implementation 
plan. It is the single authoritative summary docu- 
ment that sets forth the manner in which the objec- 
tives shall be accomplished. It defines the program 
organization, responsibilities, requirements, 
resources and time phasing of the major actions 
required. 

Program planning sets forth the development 
requirements needed to establish and maintain an 
integrated planning baseline of what is to be done, 
how it is to be done and when it is to be done. It is 
not a one-time process, since the development of 
detailed performance requirements are not estab- 
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lished at one point in time. In addition to the techni- 
cal requirements, detailed management and mission 
requirements should be established. It is a continu- 
ing process of laying out and ensuring a unified 
effort in implementing the program, adjusting to 
changing conditions, maintaining the program or 
project development plan, and integrating ongoing 
technical requirements. Although planning steps are 
laid out in a linear sequential manner, the process is 
iterative. 

The technical requirements establish the work pack- 
ages. The development of the project work break- 
down structure (WBS), consistent with the Agency 
coding structure, may also occur in conjunction with 
the planning function or it may be part of one of the 
other functions. On NASA development projects the 
WBS will normally be end-item oriented rather than 
discipline oriented. 

Resources Management 

Resources management includes the establishment, 
monitoring and maintenance of obligation and cost 
as well as the manpower baselines. Manpower con- 
stitutes the vast majority of development costs, and 
knowledge of status and trends are extremely impor- 
tant. The reporting structure for cost should be 
established and maintained with an emphasis on cost 
phasing and cost to completion. Reporting systems 
and selection of report items should be designed to 
raise questions, not to answer them; the implications 
are important, not the absolute value recorded. The 
absolute value is useful only for historical and legal 
purposes. 

The planning of reportable items is usually achieved 
through use of the Work Breakdown Structure 
accounts. The structure and analysis of report impli- 
cations should be correlated closely with schedule 
and technical performance. The recording and 
reporting of cost alone has little or no value as relat- 
ed to performance implications in the future; one of 
the main purposes of resource and schedule analysis 
is to recognize implications and to reduce manage- 
ment surprises. This allows for identification and 
evaluation of “what ifs” and alternatives. The initial 


and subsequent cost estimates must recognize and 
quantify risks and uncertainties and provide reserves 
and allowances for program changes. The require- 
ment for uncertainties and risk is as vital to project 
success as any other cost element. Having contin- 
gency funds available and using them judicially are 
integral parts of successful research and develop- 
ment efforts. 

If the contractor reporting structure attempts to 
closely parallel schedule and cost reporting mile- 
stones, extreme care should be taken that it is not 
based on the assumption of equal value milestone 
performance. This type of system can easily lead to 
some misleading assessments. If such a system is 
used, program changes can completely disrupt per- 
formance reporting and require installation of a new 
structure of report accounts and a long hiatus in 
reporting. To base a system on an assumption of 
continued program equilibrium would be a mis- 
take — uncertainty is much more likely to be the 
norm. 

Configuration Management 

The purpose of configuration management is to pro- 
vide a disciplined systems approach for the control 
of the requirements and configuration (normally 
established by systems engineering) of hardware and 
software to be developed and the process for change 
consideration. The function basically consists of 
four distinct practices: 

• Configuration Identification — The definition 
and establishment of the total technical require- 
ments (performance and functional) and the 
detailed configuration definition and documen- 
tation. Configuration identification is usually 
established incrementally as design and devel- 
opment proceed. 

• Configuration Control — The formal process 
used to establish and control changes to the 
configuration baseline. This control is effected 
through a hierarchy of formal configuration 
control boards established at the different lev- 
els of hardware and software. 
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• Configuration Accounting — Performance of 
this function “defines” the exact baseline on a 
continuing basis and provides a clear audit trail 
from the authorization of changes into the 
affected documentation. It should provide the 
single authoritative source for baseline defini- 
tion. 

• Configuration Verification — Ensures that the 
baseline configuration requirements have been 
incorporated into contracts and are fabricated 
and tested accordingly. 

Documentation Management 

Documentation management establishes data poli- 
cies and responsibilities and procedures for identify- 
ing, planning, selecting and scheduling a large vol- 
ume of data. The data management system ensures 
continual management review of NASA-generated 
and contractually required documents, eliminates 
any non-essential requirements, and assures only the 
minimum amount of documentation necessary for 
effective program management. The principal inten- 
tion of the system should be to define the informa- 
tion required, justify its need, and control the infor- 
mation after it is generated. 

Schedule Management 

This function provides for the development and 
maintenance of the master schedule and the detailed, 
interrelated schedules covering the total program or 
project to completion. It involves the requirement to 
define the schedule format, content and symbols 
used. A critical component of the function is select- 
ing the key progress indices for measuring perfor- 
mance and indicating potential problems. A system 
of reports, reviews and action feedback needs to be 
provided. Working closely with resources manage- 
ment, the analysts must evaluate performance, syn- 
thesize various inputs and implications, and generate 
and evaluate alternatives. Plans and schedules 
should provide for uncertainty and the unknown. 

The integrity, reliability and discipline of the report- 
ing system are essential. NASA should continually 


assure the end-to-end integrity of program control 
data from its source in subcontractors to prime con- 
tractors and subsequent levels of NASA. The 
importance of early problem recognition cannot be 
overemphasized: the ideal control system detects 
potential deviations before they become actual 
ones. The costliest aspect of a development pro- 
gram is time. Slippages in a program schedule are 
extremely expensive. A permanent record of all 
changes and slippages should be kept to allow trend 
analyses. 

The primary steps of management accountability — 
establishing objectives and baselines, measuring per- 
formance against baselines, analyzing and evaluating 
performance and alternatives, assigning action or 
direction, and ensuring action feedback — are applic- 
able to the management of almost any activity. 
Some aspects of control functions such as planning, 
scheduling activities, and managing resources are 
also applicable in some degree on all NASA work, 
including applied research and technology, science 
tasks, and institutional management. However, the 
collection and staffing of the full array of project 
control functions are not necessarily appropriate for 
all activities within NASA. The style of manage- 
ment and types of controls require tailoring to the 
particular objectives and problems of the individual 
activities. 

How program control functions are grouped organi- 
zationally is a consequence of a number of factors. 
Nevertheless, it is clear that all of the functions and 
their outputs need to be integrated. On a small pro- 
ject, a project manager could possibly perform the 
functions and integrate the data output. On relative- 
ly large or complex development projects or pro- 
grams, it is the opinion of the Academy team that 
management control and synthesis of program ele- 
ment progress and performance are enhanced by 
grouping the functions. A model that lays out pro- 
gram control functions suitable for most large and 
complex development projects is shown in 
Figure 13. This model assumes that program analy- 
sis is an inherent part of the functions shown. As a 
matter of preference, however, program analysis can 
be handled as a self-contained function. 
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Develop and maintain integrated planning base 
and program requirements and development 
plans; establish baselines of content, scope, 
configuration, schedule and cost; measure 
performance against the baselines; analyze and 
evaluate performance and alternatives; provide 
and monitor the procedures for changing the 
baselines; and provide the system of reports, 
reviews and action feedback. 


Establish and maintain a 
system (baseline) for a 
series of development 
plans and technical 
requirements, setting 
both the terms of 
accountability and 
performance 


Resources 

Management 


Establish, monitor and 
maintain cost and 
manpower baselines 

• Establish reporting 
and statusing 
structure 

• Correlate with 
schedule and 
performance 

• Identify and evaluate 
“what ifs" and 
alternatives 


Schedule 

Management 


Establish and maintain 

schedule baseline 

• Format and hierarchy 
of interrelated 
schedule covering 
total program 

• System of reports and 
review 

• Analyze and evaluate 
performance and 
alternatives 


Documentation 
and Data 
Management 


Establish and maintain a 
uniform system of 
documentation 


Configuration 

Management 


Formal and disciplined 
system for the 
establishment and 
control of baseline 
requirements and 
configurations of 
hardware and software 

• Configuration 
identification 

• Configuration control 
system 

• Configuration 
accounting 

• Configuration 
verification 


Program Plans 
and 

Requirements 


Program 

Control 


Figure 13. Program Control Functions. 


Current Status of Program Control in NASA 

As part of this study, the Academy made an effort to 
ascertain the current status and health of 
program/project control functions and processes 
within NASA. Interviews and discussions were held 
in both Headquarters and Centers with Center 
Directors, directors of flight projects, program man- 
agers and personnel who play roles in program con- 
trol functions. Discussions were also held with pre- 
vious NASA program directors, some aerospace 
industry officials and support contractors supplying 
management services to NASA. 

In Headquarters, the reinstitution of the Program 
Approval Document (PAD) System has not moved 
swiftly. Dale Myers, Deputy Administrator in 1987, 
sent a letter with instructions for preparation of 
PADs in June 1987. On March 14, 1989, a manage- 
ment instruction (NMI 7121.5) was issued, which 


required the specific development of 23 separate 
PADs with provisions for adding or deleting projects 
in the future. Approximately eight have been pre- 
pared and approved. The Deputy Administrator is 
holding meetings with program offices in an attempt 
to tailor the format, content and level of detail of the 
document, and to define the management processes 
to fit the desired methods of operation in an orderly 
and efficient fashion. 

Since early in its history, NASA documented its 
management policies and principles of project man- 
agement as well as instructions on planning and 
approving major research and development projects. 
These instructions were canceled in the mid-1980s 
when the PAD system was eliminated. Efforts have 
apparently been made to reinitiate or replace some of 
the canceled documents, but at this point, it has not 
been accomplished. An understanding of how the 
Agency intends to operate and what is expected in 
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terms of project management approaches and tech- 
niques does not currently exist within the Agency. A 
common concern among senior managers at the 
Centers was the apparent lack of appreciation of the 
usefulness of such policy statements on the Agency’s 
operation. 

General management status reviews continue to be 
held at Headquarters. The current review system 
provides for three separate meetings — one for Space 
Transportation and Space Station, another for all 
other programs and projects, and a third for institu- 
tional activities. According to some attendees, these 
reviews could not be characterized as disciplined 
reviews of progress against established baseline 
milestones and goals. However, program offices 
participating in these reviews do characterize the sta- 
tus and problems of projects. 

The organization and performance of program/pro- 
ject control functions within the program offices and 
the development Centers have not materially 
changed or improved since the Management Study 
Group findings in 1986 and 1988. There have been 
some changes in personnel and in the methods of 
performing the functions. One trend appears to be an 
increasing use of support contractors to provide 
some project control functions including scheduling, 
configuration management, data management, and 
elements of financial operations. The degree of con- 
tractor use varies among the Centers but the trend [in 
1989] appeared to be growing throughout NASA in 
all functions and activities in addition to program 
control. The impetus for contracting out functions 
was generally attributed to the need for supplement- 
ing the limited availability of civil servants. 
Discussion with one NASA support contractor, how- 
ever, confirmed that contractors also had the same 
difficulties in finding skilled personnel in program 
control disciplines and were faced with a problem of 
how to train their people and how to sharpen their 
skills. 

In reviewing the list of program control functions 
with NASA Center personnel, the reviewers found 
no disagreement that all of the program control func- 
tions were required and should be performed on 
development projects. Only two organizations had 


essentially all of the program control functions oper- 
ating together in one group. At Goddard Space 
Flight Center the functions were all within the 
Project Director’s office reporting to the Deputy 
Director for resources. Scheduling, configuration 
management and data management functions were 
performed by a support contractor and were under 
civil service monitors responsible for the functions. 
A discrete function for project planning was not 
within the project offices. The Space Station office 
at Johnson Space Center (JSC) is the other organiza- 
tion having a fairly complete grouping of functions 
under the program control division. In the other pro- 
gram offices at JSC, program control functions are 
not integrated in one group but are being performed 
in one way or another in various organizations. 

At the Lewis Research Center, steps have been taken 
in the Space Station project to integrate resource 
management, scheduling, and configuration manage- 
ment in a program control organization. At the 
Marshall Space Flight Center, there is a fairly con- 
sistent pattern of combining scheduling and 
resources management in a single organization in the 
project offices. Except for the cases noted above, the 
remainder of the NASA Centers and the 
Headquarters program offices do not have organiza- 
tionally integrated program control functions. The 
functions are either not performed, are scattered in 
various subgroups, or are done informally. 

An Agency cost estimate is always prepared on new 
development projects prior to evaluation and selec- 
tion of contractors. However, there does not appear 
to be a uniform procedure for recycling and validat- 
ing new estimates after selecting the development 
contractor. Rather, the contractor’s negotiated bid 
generally becomes the baseline against which any 
changes are incrementally made. This is true even 
though the contractor’s estimate is usually consider- 
ably lower than the government’s estimate. The 
rationale for the government’s higher estimate in 
most cases is quickly forgotten. Credibility begins to 
be attached to the contractor’s estimate, which is nei- 
ther justified nor borne out by history. Since it takes 
some time for deficiencies to become apparent, they 
generally come as surprises and result in more cost- 
ly schedule slippages. In too many cases a large pro- 
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portion of the time available to the staff of project 
resource and schedule management groups is spent 
on finding near-term funding solutions to these “sur- 
prises.” 

As a general observation, too little effort is spent by 
both resource and schedule groups in analyzing 
potential problems or risks and in selecting critical 
reportable milestones that could give some advance 
notice on the probabilities of problems. Close corre- 
lations of reportable schedule and cost performance 
data is desirable, but the critical indices of perfor- 
mance are not always precisely aligned with a hard- 
ware-driven work breakdown structure. 

There is an apparent lack of emphasis on laying out 
logic diagrams or networks on projects, particularly 
prior to selecting schedule and resource reporting 
items. The researchers know of no better way to 
comprehend interrelationships and interfaces of 
efforts on components, subsystems and systems. 
When these networks are laid out in time sequence, 
critical schedule and resource reporting indices 
become much more apparent and risks are easier to 
assess. The special virtue of logic diagrams is that 
they allow planners to incorporate time, resources, 
and technology into strategies, thus linking temporal 
horizons with contextual changes. 

As a generalization, reviews at the Centers appear to 
be more structured toward the assessment of project 
performance than they are at Headquarters. Many of 
the reviews at the contractor plants, however, seem 
to be primarily scheduled visits with fixed agendas, 
and with large groups spending great amounts of 
time looking a viewgraphs. It was not apparent how 
often site visits by project control personnel were 
made for the purpose of assessing performance and 
verifying the integrity of reported data at its source. 
Regardless of how scientific the approach or how 
sophisticated the management system and tools are, 
there is no substitute for a simple visual assessment. 
Coordination of those supplying performance data is 
essential. 


Training 

Traditionally within NASA, program control person- 
nel have gained skills and knowledge through first- 
hand experience and from their experienced supervi- 
sors. Immersing themselves in program/project 
research and development activities is still the most 
common way of gaining project management knowl- 
edge. Forming mentor relationships — working with 
a person who can provide counseling, guidance and 
advice — is also used to gain the skills and credentials 
of program control. However, experienced program 
control personnel are becoming fewer within NASA. 
According to interviewees at the Goddard Space 
Flight Center, in the past, many program control staff 
first studied operations research or industrial engi- 
neering, then acquired on-the-job skills and subse- 
quently passed on lessons learned by various means. 
Rarely did program control staff receive formal 
training related to specific functions such as the 
establishment and maintenance of a configuration 
control or scheduling system. 

NASA and contractors currently face difficult prob- 
lems in recruiting experienced program control staff 
due to a number of reasons, from limited career paths 
to elimination of industrial engineering disciplines at 
many major universities. As mentioned earlier, in 
response to recommendations from the Phillips 
Committee, NASA decided to formalize efforts to 
help in the development and training of managers, 
including program control personnel. Formal train- 
ing will be provided in such areas as resources man- 
agement, schedule management, and configuration 
management. Analytical skills and the philosophical 
and logical foundations of program control, howev- 
er, cannot be learned just by attending classes. They 
require application and the achievement of an end 
result as well. Self organization, program interest, 
ability to coordinate individuals and data, a ques- 
tioning attitude, resiliency, sensitivity, imagination, 
and practicability are other nonemperical qualities 
that are valuable in program control work, but are 
beyond the realm of classrooms. In sum, formal 
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courses can only complement, not replace, hands-on 
experience and the inherent qualities of key person- 
nel. This is because analytical skills are, to a large 
extent, embodied in people and institutions, not just 
in physical objects like computers. 

It is anticipated that formal development training 
will be provided by both civil servants and contrac- 
tors. There will be a core curriculum which will be 
designed to serve business, technical and program 
and project management staff as well as a series of 
detailed courses designed for people who will be per- 
forming functions in specific areas. It is expected 
that the importance of integration of the program 
control functions and synthesis of data, personal 
responsibility and accountability, and disciplined 
procedures will be stressed. How the courses are 
structured and how consistent they are with the past 
experiences and needs of trainees will have a strong 
bearing on the prospects for success of the training 
efforts. Equally important, however, will be the sup- 
port of top management at the Centers and 
Headquarters. Their interest will have a serious 
impact on the outcome of the project. If top man- 
agement is sensitive to and supportive of the need for 
training and displays a strong commitment to the 
training program, the probability of success increas- 
es tremendously. Perhaps more significant is that if 
top management is involved and accurately commu- 
nicates its involvement, the entire effort will be per- 
ceived as credible and worthwhile. 

Recommendations and Observations 

NASA has successfully managed some of this coun- 
try’s most complex technology and development 
programs. These successes have included the appli- 
cation of sound program control processes. The 
basic concepts of program management and program 
control have not changed, although computerized 
systems have the capability to enhance the quality 
and effectiveness of documentation, communica- 
tions, evaluation tools and support systems. Much of 
the new capability of tools and support systems have 
been incorporated in NASA, but over time NASA’s 
use of the basic management control disciplines has 
weakened. Strengthening program control involves 
the improvement and utilization of certain disci- 


plines, the existence of a conducive Agency environ- 
ment and an understanding throughout the Agency of 
the leadership’s policy and objectives. The follow- 
ing recommendations are oriented toward improve- 
ments in program control processes and practices. 

Enhancement of Agency Environment for 
Effective Program Control 

This study concludes that is would be extremely 
helpful for NASA personnel to be aware of the 
importance attached to program control functions be 
the Office of the Administrator. This awareness can 
result in the reinvigoration of program management 
disciplines throughout the Agency. An effective 
method of informing Agency personnel and contrac- 
tors would be through appropriate issuances setting 
forth Agency intentions for conducting its business, 
expectations of all elements and policies and proce- 
dures for program/project approvals, assignment of 
responsibilities and the explicit accountabilities of 
organizations and individuals. The following actions 
would be helpful: 

• Issuance of Agency policies and processes for 
the approval and conduct of projects, the 
assignment of responsibilities and the terms of 
accountability of organizations and personnel. 

• Establishment of regular performance reviews 
against approved baselines of development 
plans, schedules and cost appropriate for this 
level of management. 

• Facilitation of rapid communications to and 
from all NASA elements regarding program 
control functions, tasks and feedback on action 
assignments. 

Development of Training Activities for Program 
Control 

The primary emphasis should be on understanding 
the role of program control functions in relation to 
and in context with the program/project manager and 
other groups and functions of the program office, par- 
ticularly systems engineering. Systems engineering 
includes those activities required to transform mis- 
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sion needs into a comprehensive and definitive set of 
systems performance requirements. It also includes 
the activities needed to define a preferred systems 
configuration and its detailed performance require- 
ments. The results of these activities set much of the 
baseline detail for program control functions includ- 
ing program plans and configuration management 
and parameters for schedule and cost management. 

Program control is the total management process of 
establishing and maintaining the official develop- 
ment plans and program baselines in a manner which 
maximizes success in meeting a program’s overall 
objectives. Although the following topics are not all- 
inclusive, some suggested program control training 
activities are (more detail shown in the Appendix): 

1 . Philosophy, content and context of effective pro- 
gram/project control. 

2. Planning and documentation of requirements. 

3. Content and processes of configuration manage- 
ment. 

4. Logic diagrams or networks. 

5. The scheduling function and process. 

6. Basics of primary methods of cost estimating. 

7. Resource management and control. 

8. Presentation of data. 

The most important element in evaluating and 
assessing the status of a project and providing pro- 
gram control is the understanding of the objectives, 
technical content, development approach, and the 
interrelationships and interfaces involved in its 
development. Throughout this report, the Academy 
researchers have taken the position that program 
control is not a collection of the separate functions 
that comprise it, but that it is an understanding of the 
plans and approach and the interrelationships of the 
functions and performance of configuration, sched- 
ule and resource management. 


The most meaningful implications from performance 
data cannot be drawn from the independent func- 
tions, but rather, only when the data are integrated. 
For this reason we have emphasized the integrated 
understanding of the roles rather than skills and 
tools. Tools and skills can be very important but 
only when one understands their limitations as well 
as advantages and knows when they can be usefully 
applied. In this context, emphasis on skill training is 
important with regard to particular tasks such as 
logic networks, a means of focusing data for the 
maximum information output and the presentation of 
interrelated performance data. 

Observations 

Until conducting this study it had not been apparent 
to the researchers the degree to which NASA has 
become staffed with support contractors as opposed 
to career civil servants. Onsite contractors appear to 
now exceed civil servants. The impact of this condi- 
tion potentially can have serious consequences on 
NASA’s program management and control capabili- 
ties. 

As stated earlier in this report, NASA projects push 
technology beyond the current state of the art. 
Traditionally NASA has had the civil service and 
fabrication capability in its Centers to conduct the 
appropriate depth of studies, examine objectives and 
missions, develop the technical concepts for accom- 
plishing missions, determine feasibility, and provide 
the conceptual design. If it was decided to budget 
and contract for the design and development of a 
project, the inhouse capability existed to manage, 
technically monitor, evaluate and direct such con- 
tracted work. If technical problems arose at the con- 
tractors’ plants, the capability existed to help provide 
solutions and correct the problems. Some of the 
major objectives of program/project control are the 
early identification of potential problems, avoidance 
of surprises, provision of workarounds, and the abil- 
ity to obtain help in providing solutions. This pre- 
cept of the importance of early problem identifica- 
tion assumes the availability of the technical capa- 
bility to participate in solutions to such problems. 
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Funding pressures on projects have continued since 
the early 1970s, and less funding allowance for the 
contingencies of the “unknown” has been the result. 
As surprises occur and additional funds are not 
available, schedules usually become the variable on 
which short-term solutions to fiscal year funding 
problems are based. The obvious result is an 
increased run-out on total cost and shrinking credi- 
bility. 

With the increasing contractor staffing, NASA engi- 
neers have less and less “hands on” experience. 
Service contractors are increasingly being used at 
Centers to perform project control functions such as 
scheduling, configuration management and elements 
of financial management. In effect, this is using con- 
tractors to monitor the performance of prime devel- 
opment contractors. This situation is leading some 
NASA managers to question the Agency’s continu- 
ing ability to manage contracted projects and control 
costs. 

NASA remains responsible for the performance of the 
work, but with a reduction in capability to influence 
and correct performance. How well the Agency meets 
demands relating to program performance has a major 
effect on its ability to effectively run programs. 

Appendix 

Suggested TVaining Activities 

The following topics are not inclusive in the sense 
that they cover all items. 

1 . Philosophy, content and context of effective pro- 
gram/project control. 

• What is meant by “control”? 

• An explanation of how the main functions 
relate to each other. 

• Importance of understanding the totality of the 
project. 

• Importance of understanding interrelationship 
of elements and interfaces. 


• Importance of ensuring integrity of reported 
data to source level. 

• Importance of concentrating on the implica- 
tions of reported data rather than on the factual 
data. 

• Anticipation of development difficulties and 
changes in external environment. 

• Continual assessments of “what ifs.” 

• Importance of a questioning approach. 

• Requirement for disciplined processes and pos- 
itive monitoring. 

• Barriers to effective program control. 

2. Planning and documentation of requirements. 

• Importance of maintaining development plan 
baseline. 

• The necessity of a series of subsidiary plans, 
actions and schedules. 

• Documentation of requirements. 

• Technical and program reviews and results. 

3. Content and processes of configuration manage- 
ment. 

• Importance of early development and docu- 
mentation of configuration requirements and 
preparation of a configuration maintenance 
plan. 

• The systematic approach of defining and docu- 
menting the detailed configuration. 
Understanding of the need for incremental 
identification as design and development pro- 
ceed. 

• The significance of positive control of changes 
to configuration. Importance of evaluating 
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impact of individual proposed changes on 
operational capability and total cost. 

• Importance of clear audit trail of changes and 
maintenance of the exact baseline. 

• Necessity for effective verification that base- 
line configuration has been implemented. 

4. Logic diagrams or networks. 

• Understanding how to develop networks. 

• Importance for understanding the total job and 
the interrelationship of the components of the 
job. 

• Relevance of networks to effective analysis 
and synthesis of performance data. 

5. The scheduling function and process. 

• Critical importance of identifying known and 
potential development risks. 

• Planning for the unknown. 

• Understanding interrelationships and interfaces 
of development processes and organizations. 

• Importance of selecting the critical indicators 
of progress or problems — the most important 
scheduling function. 

• Identification of indicators as far “upstream” as 
possible from critical progress points. 

• The danger of becoming mesmerized with sys- 
tems. The need to understand weaknesses bet- 
ter than positive elements and to keep systems 
as simple as possible. 

• The amount of time required for administrative 
and decision processes. This time requirement 
cannot be overlooked. 

• The costliest aspect of a R&D project is time. 
Slippages are extremely expensive. 


• Emphasis on early problem recognition. 

• Importance of having only authenticated and 
dated schedules. 

• Maintenance of permanent record of all 
changes and documentation of slippages. 

6. Basics of primary methods of cost estimating. 

• Understanding of concepts, processes, when 
each is most useful, advantages and disadvan- 
tages: parametric cost estimating, analogy esti- 
mates, engineering estimates (“grassroots,” 
“bottom-up”) and expert opinion or Delphi 
techniques. 

• Dangers of accepting contractor’s negotiated 
cost estimate without complete reverification. 

• Importance of quantifying risks. 

• Importance of provision for and use of 
reserves. 

• Risks involved in using cost goals as incentives 
in cost estimating and the use of “design to 
cost” concepts on R&D operational systems. 

7. Resource management and control. 

• Establishment of a cost reporting system. 

• Importance of correlating manpower reports on 
R&D projects. 

• Importance of integrating cost data with sched- 
ule performance. 

• Verification of end-to-end integrity of data 
reported. 

• Understanding the contract structure, and 
nuances of differences in definitions and accu- 
mulation processes of prime and subcontrac- 
tors. 
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• Importance of onsite verification of data and 
calibration of personnel supplying data. 

• Reporting of data should raise questions, not 
answer them. 

• Trend analyses. 

• Emphasis on run-out and cost to completion. 

• Importance of continual work on “what ifs.” 

8. Presentation of data. 

• Determination of objective or purposes of pre- 
sentation: What is the message or information 
to impart? 


• Determination of desired outcomes. 

• Avoidance of reams of cost, schedule or engi- 
neering data. The need to focus presentations 
and use only data which contribute to under- 
standing context, significance and implications 
of information. Detail can overwhelm strategic 
choices. 

• Factual data may or may not be significant to 
future actions or decisions even though they 
may be important for legal or audit purposes. 

• The need to sequence messages in a priority, 
logical or temporal order. The use of unam- 
biguous language. 
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Resources for NASA Managers 


by Dr. William M. Lawbaugh 


Book Reviews 

Mana gin g in a Time of Great Change 

by Peter F. Drucker 
Dalton: New York, 1995. 

In 1946, Peter Drucker redefined employees as 
resources instead of expense or cost items in Concept 
of a Corporation. Post-war Japanese reformers 
adopted him as their business guru and guide, and 
The Practice of Management (1954) took Europe by 
storm. Due largely to his influence, institutions 
began to re-organize around the flow of things to the 
flow of money and now to the flow of information. 

Like previous Drucker books such as The Frontiers 
of Management (1986) and Managing for the Future 
(1992), this book was “pre-tested” chapter by chap- 
ter in magazines such as The Atlantic Monthly and 
Harvard Business Review. It lacks flow and conti- 
nuity, but the insights are certainly worth pondering. 

For example, he says: “The current emphasis on re- 
engineering is from the flow of things to the flow of 
information. The computer is merely a tool in the 
process.” Post-capitalist executives are knowledge- 
workers who must figure out what information is 
needed and, “most importantly, what they do not 
need.” 

Among his “five deadly business sins,” Drucker 
includes “feeding problems and starving opportuni- 
ties.” He calls problem solving mere “damage con- 
tainment” and says only opportunities will produce 
measured growth and tangible results. He has six 
rules for U.S. Presidents, including: “Concentrate, 
don’t splinter yourself,” but recent all-out efforts to 
achieve universal health care or gay rights in the mil- 
itary seem to have fizzled. 

From his perch in academia (Claremont Graduate 
School), Drucker can speculate on “The End of 


Japan, Inc.?” and “really” reinventing government, 
but management, not political science, is his forte. 
He does admit, however, that two answers have been 
wrong this century in dealing with social need. The 
first answer was to let government solve social prob- 
lems, but “society is becoming sicker rather than 
healthier.” The second wrong answer was formulat- 
ed in his 1942 book The Future of Industrial Man , 
that the corporation became a worker’s “community” 
from cradle to grave. However, “entitlements” and 
“fringe benefits” are not his solution today. Rather, 
echoing his Managing the NonProfit Organization, 
written a half-century later, Drucker proposes: “It 
profits us to strengthen nonprofits” such as AA, 
parochial schools and private relief agencies to 
address our social ills most effectively. 

Peter Drucker is on more solid ground writing about 
management. In team-building, he clearly prefers 
what could be called “basketball” where few players 
mold and work together quickly, such as at GM’s 
Saturn Division. Detroit and most American indus- 
tries were built on the sluggish, inflexible “baseball” 
team model, while Japan was more like “football” 
where the boss or coach still called all the plays. 

As for the “Change” in the title, Drucker says, “For 
managers, the dynamics of knowledge impose one 
clear imperative: every organization has to build the 
management of change into its very structure.” He 
suggest three ways to do this: continuous improve- 
ment of product, self or service; exploitation of suc- 
cessful knowledge (new products, selves or ser- 
vices); and organized, systematic innovation — every 
organization’s necessary core competence. 

Education and School are at the epicenter of 
Drucker’s new information-based society for knowl- 
edge workers. Yet, he says, “Management, in most 
business schools, is still taught as a bundle of tech- 
niques,” such as budgeting and planning. As impor- 
tant as there are, Drucker says, it is far more impor- 
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tant in this age to develop “competencies,” like 
working under pressure, learning how to learn, 
knowing what to know, and being able to gather 
organize and present useful information. When 
Drucker says “we need to measure, not count,” he 
means moving away from traditional cost accounting 
to looking at value, quality and investment. “The 
key is not ‘cost’ but ‘cost-effectiveness.’” 

In this competency-based education environment, 
the knowledge worker (a term coined by Drucker in 
his 1959 book The Landmarks of Tomorrow) requires 
“a habit of continuous learning.” Thus, for Drucker 
at least, management is one of the liberal arts instead 
of a social science. It is not “experience-based” but 
rather “learning-based.” Core competencies lead to 
“being able to do something others cannot do at all 
or find difficult to do even poorly,” which should be 
enough to carry us to the end of his predicted social 
transformation in 2010 or 2020. 

Multimedia for Decision Makers 

by Jeff Burger 

Reading, MA: Addison-Wesley, 1995. 

Managers and executives often wonder how their 
communications can reach more people and become 
more effective. “Multimedia” is the suggested 
answer, but the decision maker needs to know how to 
integrate the various media (text, graphics, audio, 
video, interactivity) in the office and make it cost- 
effective. That’s where Jeff Burger’s new book 
comes in. 

Multimedia for Decision Makers is an overview of 
multimedia applications for managers, not techni- 
cians. It is conceptual rather than technical, and it 
affords a basic grasp of the possibilities and benefits 
of using more than one medium in presentations, 
trade shows, direct marketing, information manage- 
ment, training and teleconferencing. 

“Interactivity” is the key in space-age communica- 
tions, according to Burger. It is often noted that we 
grasp 20 percent by hearing, 40 percent by seeing or 


reading, and a whopping 80 percent by doing. 
Interactive multimedia enhance our communications 
immensely, especially through the Internet and CD 
ROM technologies. 

Just as the Internet was at first a Cold War effort to 
sustain bomb-proof communication, laser discs and 
CD-ROMs were first used in military training, says 
Burger, such as interactive learning for nuclear sub- 
marine management, in place of bulky service man- 
uals. Electronic kiosks incorporating graphics, 
sound, modem transmission and vending are being 
developed in California for everything from bill pay- 
ment and driver’s license renewal to state lotteries. 
Space travel is made much more exciting (and edu- 
cational?) through interactive multimedia simula- 
tors. Some call it “edutainment.” 

Edutainment could soon involve videos and music 
on demand, smart” games, computer-assisted 
research, interactive fiction adventures and even 
home based shopping comparison, depending on 
passage and implementation of new telecommunica- 
tion legislation. As Burger points out, “throughput is 
only as efficient as that of the smallest artery.” In 
other words, one burst of interactive multimedia col- 
lapses when the fiber-optic cable feeds into a mere 
copper line on your street. 

What Burger does not point out is that much of this 
“new” technology has been around for a long time, 
but there has been little or no consumer demand for 
it. Bell Labs, for example, introduced the 
Videophone in the era of the Kelvinator, but con- 
sumers preferred better food storage over showing 
up on the telephone. The first facsimile transmission 
was sent from Lyons to Paris in 1865, but no one 
seemed to need it until recently. The USPS has aban- 
doned its plan for user-friendly postal kiosks. We 
still do not need or want the Videophone, apparently. 

Nevertheless, Burger’s books presents at least a 
dozen alternatives to the typical viewgraph presenta- 
tion, all of them feasible and economical. 
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Silicon Snake Oil 

by Clifford Stoll 

New York: Doubleday, 1995 

Subtitled “Second Thoughts on the Information 
Highway,” this book fills a void: “there’s damned 
little critical discussion of the implications of an 
online world ” Is the Internet oversold? Do networks 
deliver on education? Progress? Is this the ultimate 
revenge of the nerds? 

Clifford Stoll is a planetary astronomer by training 
who is offended by colorized and computer- 
enhanced images of outer space sent online by 
NASA, for example. He finds them fraudulent, 
“since infrared images have no color,” he says. He 
also finds computers in the classroom “expensive 
and semi-reliable,” providing only flat, black-and- 
white, one-dimensional info. They are, too him, as 
useless as television. Books (like his?) do a better 
job. “Learning is not easy,” he declares, excoriating 
“edutainment” devices. 

Multimedia? “Wrong, since there’s only one medium 
employed: the computer.” 

Interactive? Nope, all the outcomes are, of course, 
preprogrammed. “The experience is about as inter- 
active as a candy machine.” 

Eye-hand coordination, at least? A neighborhood 
game of soccer is far healthier, and a box of crayons 
and a big sheet of paper far more expressive. 

Educational? Researchers and creative folks publish 
their best stuff in journals, and the gold online is hard 
to distinguish from all the dross. Besides, CNN will 
keep you better informed than the Internet. 

A virtual community? Yes, but how impoverished 
without a church, a cafe, a theatre, a museum or even 
a corner bar. “And no birds sing,” he adds. No chil- 
dren, no hearth, no warmth. 

Great jobs? “Well, no. Computer skills no longer 
guarantee employment,” he says. “Programming 
jobs are easily exported,” like hardware manufactur- 
ing and software piracy. 


Telecommuting? Talk about turning home with all its 
distractions, into a prison, he asserts. And tell that to 
your dentist or auto mechanic. 

Email? Stoll finds faxes are cheaper, faster, better — 
and more reliable, secure and universal, and with no 
junk. Real (snail) mail is more personal and warm. 
Telephone, too. He’s met dozens of teenage com- 
puter wizards who have never written a thank-you 
letter. 

Clifford Stoll is not your average troglodyte, Luddite 
or computer dubunker. He was an Arpanet user long 
before we had an Info Highway, and his first book, 
The Cuckoo's Egg, is all about how he nabbed a 
German spy ring on the Internet, which he now calls 
“that great digital dumpster” of disconnected data. 

The biggest loser in the online culture is the library 
as we know it — an organized set of books and peri- 
odicals. Yet, libraries are strapped because they have 
had to invest in computer systems and software that 
are soon obsolete. (Look at their earlier investments 
in punch-card and paper- tape readers, reel -to reel 
tapes, 78 rpm disks, 8 mm. movies, 8-track tapes, 
and new books on tape, CD-ROMs, ASCII files, 
FORTRAN, Basic, Word 2.3, etc.). Their hours are 
shorter but wisdom is diminished. 

Stoll distinguishes between wisdom and data. 
Online you can find plenty of data (like drinking 
from a firehose), little usable information, less 
knowledge, and hardly any wisdom, since nearly 
nothing before 1980 is digitized. Besides, who 
would really prefer to read a book (or periodical) off 
an LCD or CRT instead of real paper? 

The Leadership Challenge (2nd. edition) 
by James M. Kouzes and Barry Z. Posner 
San Francisco: Jossey-Bass, 1995. 

Tom Peters, in the Foreword to the second edition of 
this thick (400-plus pages) new book, says: 
“Management is mostly about ‘to do’ lists (can’t live 
without them!)” but “Leadership is about tapping the 
wellsprings of human motivation.” The ’90s version 
of that ’60s word appears to be “empowerment.” 
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Posner and Kouzes describe how their world has 
changed in the past seven years since the first edi- 
tion. Power has shifted from a master-slave business 
hierarchy to a flattened client-server of empowered 
people. Like Peter Drucker they believe knowledge 
is the new currency, replacing land and capital. They 
see less loyalty and workforce commitment but also 
less job security and more self employment, by 
choice or not. They also see, surprisingly, a renewed 
search for meaning and suggest that leaders become 
“more like trusted friends in this increasingly cynical 
world.” Paradoxically, the authors say: “We’re all 
connected” in a global village, but also: “The world 
is disconnected” with more countries, more products 
and more services in a marketplace of smaller pieces. 
They ask: “How can a leader unite such a diverse 
and disparate constituency?” 

Simple. Just apply the same “five fundamental prac- 
tices of exemplary leadership” and the adjoining “ten 
commitments of leadership” found in the first edi- 
tion, but with new lingo, fresh anecdotes and new 
“personal best” case studies. In brief, here are the 
practices and their dual commitment, upon which the 
entire book is based: 

1. Challenge the status quo by embracing change 
and innovation, and by taking risks, but be will- 
ing to accept and learn from any resulting mis- 
takes. 

2. Inspire a shared vision. Dream of an ideal future 
but also set others on fire by communicating the 
vision clearly and vividly. 

3. Enable others to act by building trust while giv- 
ing power away. 

4. Model the way by personal example that is con- 
sistent with shared values, and build team com- 
mitment with frequent small wins. 

5. Encourage the “heart” of subordinates by cele- 
brating team accomplishments and by recogniz- 
ing individual contributions. 

“Love — of their products, their services, their con- 
stituents, their clients and customers, and their 


work — may be the best-kept leadership secret of all,” 
say the authors. 

Actually, these five principles are the opposite of the 
way traditional management operates. Most bosses 
will expect employees to fall in line and make things 
run like clockwork, but Item One calls for defiance 
to the status quo, shaking up the organization. 
Traditional management may tend to focus on the 
short term if not present moment, but Item Two 
gazes well into the future. Item Three has the leader 
divest of power while traditional management may 
seek rather to consolidate it. Cool and aloof tradi- 
tional management behind closed doors may try to 
rule by threat and fiat, but Item Four suggests lead- 
ing by personal example. Item Five would be sound- 
ly denounced by control freaks as sentimental hog- 
wash, but Posner and Kouzes’ leader serves and sup- 
ports instead of command and control. Tom Peters 
even goes out of his way to say that Jim Kouzes, 
“like Winston Churchill, cries easily; he cares.” 

However, the authors present 36 pages of theory and 
evidence of statistical methodology and scholarly 
footnotes to prove they are not sentimental old fools. 
Kouzes served in the Peace Corps and Posner sits on 
the local board of Big Brothers/Big Sisters. Together 
they also authored Credibility: How Leaders Gain 
and Lose It, Why People Demand It (1993). The 
subtitle of this book reads: “How to Keep Getting 
Extraordinary Things Done in the Organization.” 

Dive Right In, The Sharks Won’t Bite 

by Jane Wesman 

Dearborn: Financial Publishing, 1995. 

Although Jane Wesman ’s new book is subtitled “An 
Entrepreneurial Woman’s Guide to Success,” it is 
chock full of good tips and advice for project man- 
agers of both genders. The first three chapters focus 
on getting started in a new business, but the other 13 
chapters are filled with generous advice from a real 
pro. 

Jane Wesman was a publicity director for New York 
publishers before she started her own public rela- 
tions firm 15 years ago. From experience, she says 
the entrepreneur needs courage, determination and 
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energy to survive in a tough market. “Energy is key,” 
she says, urging a low-fat but nutritious diet and rig- 
orous exercise to “clear your head and think cre- 
atively.” Being well groomed also instilled confi- 
dence. 

For a woman, access to capital or start-up loans is the 
biggest problem. She started at home by lining up 
clients first and securing advances, but today she 
could have tried a small business “incubator,” a suite 
of offices with common reception, telephone, fax 
and copier, usually connected to a university or local 
(county or state) government. She was wise enough 
to shop around for the right lawyer and accountant 
for “a good fit” before she retained them. 

She spent a lot of time hiring just the right employ- 
ees, too. Most new hires were cheerful and upbeat; 
none of them was hired just for the money because 
they would leave just as soon as a competitor offered 
more. Generous benefits and incentives were 
offered in lieu of more money. 

To fight the “sharks” in the old boys’ network, 
Wesman joined women’s clubs and organizations as 
networking venues. She returned every phone call, 
and she never held grudges; people appreciated her 
thoughtfulness and often recommended her firm to 
others. She offers the reader 18 tips in the final chap- 
ter, her favorite, ending with “Be gentle with your- 
self . . . Think about what makes you special and 
what brings joy into your life.” 

Perhaps her best advice is her first “sharkproof strat- 
egy for success.” Keep a journal, she says. Record 
your feelings and impressions. The private journal 
becomes her lessons learned. 

From a colleague at Harvard Business school she 
learned and kept “the notebook system.” Buy one of 
those marble notebooks, like grade school kids use, 
the one you cannot tear pages from due to the thread 
binding. List all the things you need to do on the 
right hand page, and put meeting notes, reminders 
and phone numbers on the left hand page. Like the 
journal, this becomes a valuable record for retrieval 
and reflection. 


The Road Ahead 

by Bill Gates 

New York: Putnams, 1995. 

“Human history becomes more and more a race 
between education and catastrophe,” wrote H.G. 
Wells in 1920. Seventy-five years later, Bill Gates of 
Microsoft, arguably the wealthiest man in the world, 
holes up in his summer cabin to bang out a draft of a 
book on his PC in order to begin a dialogue on the 
information superhighway, highway or road. He’s 
not sure. 

If he were sure, we should all go out and buy his 
book and invest in the stocks and commodities he 
deems hot. No need to. The Road Ahead is surpris- 
ingly simplistic, if not a bit self-serving. 
Nevertheless, when a guy like Bill Gates or E. F. 
Hutton speaks, we should no doubt give a listen. 

No single theme holds this book together. It is part 
biography, part polemic and in large part pure spec- 
ulation. He denounces the appropriateness of the 
term “information highway” in the Foreword, but 
uses it uncritically anyway throughout the book. 

In essence. Bill Gates agrees with H.G. Wells — edu- 
cation is the best, perhaps the only solution to the 
bumps and potholes as we ride the information high- 
way. Education will reduce our fears of emerging 
technologies and will enable us to navigate better the 
road ahead. 

Education to Gates, however, does not mean formal 
schooling. To him it means tinkering, serendipity, 
cramming. His biodata is revealing. His best friend 
(and later business partner) was three years older and 
able to explain to inquisitive Little Billy how gaso- 
line was made. Later, the teenage hackers with 
pocket protectors read Popular Electronics and got 
hooked on the Altair (a Star Trek destination) 8800 
minicomputer and wrote a language (Basic) for it — 
the rest is history. At Harvard, Gates cut most of his 
classes and just crammed for the final exams. The 
rest of the time was spent developing software and 
then Microsoft. He dropped out of college at age 1 9. 
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Nevertheless, “education” has more hits in Gates’ 
book than any other topic. His main purpose in writ- 
ing the book is to educate, as a travel guide to the 
road ahead. If he were the businessman in Mike 
Nichols’ film “The Graduate,” his one-word bit of 
advice to Benjamin (Dustin Hoffman) would not be 
“Plastics” but rather “Information.” 

Down the road ahead, Gates sees convergence of 
television, the computer, cable and telephone into an 
“interactive media server” for home entertainment 
and telecommuting business. Out on the road he will 
carry the “wallet PC” that not only dispenses “digital 
money” but also sends and receives faxes, email, 
stock reports and games. It connects to Global 
Positioning System (GPS) satellites. 

Conveyance of media depends, of course, on 
telecommunications reform, and wireless technology 
subsumes the Internet somehow. The CD-ROM, 
however, is praised for its here-and-now potential. 
He unabashedly plays the “Encarta” encyclopedia 
disk (brought to you by Microsoft), but the real test 
will be the CD-ROM that comes with the book. Will 
“readers” discard the book and pop in the disk 
instead? After all, that little Road Ahead CD-ROM 
contains every word of the book plus an “interactive” 
tour of the highway in business, home and school, in 
brief video and briefer audio selections. There’s 
even an “Ask Bill” application, showing an animat- 
ed Bill Gates sputtering glittering generalities. A 
totally useless “web browser” connects you to sam- 
ple a commercial online service like CompuServe IF 
you have a modem. If you don’t have “Windows 
95,” forget it. If you have a Mac, forget even the 
CD-ROM. 

In fact, most of The Road Ahead is forgettable. His 
“Implications for Business” are neither fresh nor 
original, his notions on “Friction-Free Capitalism” 
are pie in the sky, and the last chapter, “Critical 
Issues,” covers issues that are not critical at all. His 
attempts to arrive at a pricing policy for intellectual 
property are as important as the government’s 
attempt to tax the Internet. 

Nevertheless, The Road Ahead is an easy read (or 
view, if you use the CD ROM) and mildly interesting 


(especially when he tells the history of Microsoft or 
the story of his new house), but the Nov. 29, 1995 
Newsweek cover story and pictures are better edited 
and the October National Geographic much more 
informative. 

Video Reviews 

The Upper Atmosphere Research Satellite 

(UARS) Mission 

with Charles Trevatan 

Goddard Space Flight Center, 1992. 

The Upper Atmosphere Research Satellite (UARS) 
Mission is described in this 45-minute video as 
“tremendously successful” by narrator Lee Blasso. 
It was delivered two months early, $30 million under 
budget, and all systems functional properly in orbit. 

The PPMI Lessons Learned and Shared Experiences 
video features the last project manager, Charles 
Trevatan, who took over in 1991 from Peter Burke 
when he became Deputy Director of the Goddard 
Space Flight Center. 

The September 12, 1991, launch marked the begin- 
ning of NASA’s Mission to Planet Earth. The space 
observatory was to study the Earth’s upper stratos- 
phere and mesosphere for ozone depletion during an 
1 8-month mission. 

Trevatan said the good news was cost control in 
addition to performance and schedule, but especially 
“dedicated people and organizations.” Deputy 
Project Manager John Donley agreed, noting “stabil- 
ity of people” in this 11 -year project that began in 
1980, especially the scientist investigators. Of the 
ten science proposals accepted in 1978, eight of them 
flew. 

Dr. Carl Reber, Project Scientist, added that the most 
important aspect was mission philosophy: that this 
was a scientific mission with the end-product as sci- 
ence. A well-defined set of requirements assured 
success a decade later. 

Trevatan noted that since this was a multimission 
spacecraft, the project showed cost savings up front. 
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Also, “we knew the interfaces right off,” he said, 
referring to thermal, mechanical, etc. 

Richard Baker, Deputy Project Manager for 
Resources, said the UARS had an adequate flow of 
funds throughout the lengthy project, partly stimulat- 
ed by the passage of the Clean Air Act. With good 
control of contract modification and requirements, 
along with good interface integration schedules, “we 
were able to avoid downtime with a full pipeline.” 

Ellen Herring, Data Systems Manager, noted the dif- 
ficulty in trying to coordinate 20 remote analysis 
computers in the U.S., France, Canada and England. 
However, the team focused early on data system 
activity and gave a three-day stress test for data 
delivery bottlenecks. She found the “training mater- 
ial too difficult to comprehend” and recommended 
“modular training” as users are phased in. 

The “tremendously successful” project was not per- 
fect. The ISAMS founder from Oxford University 
failed, due to bad lubricant in the bearings after a 
change in motor type and circuitry. Also, a motor 
clutch stuck on orbit after eight months. Trevatan 
calls this systems failure a “design flaw” rendering 
the motor commendable but not automatic. 

This video was narrated by Len Blasso. Judy Grady 
Hamburg was producer, director and scriptwriter for 
Media Specialist Associates. Gene Guemy served as 
NASA Technical Monitor. 

The Cosmic Background Explorer 

with Roger Mattson 

Produced by Technical and Administrative Services 
Corporation, TADCORPS, June 17, 1991. 

“Lessons Learned in the COBE Project” was pro- 
duced shortly after the highly successful launch and 
early scientific data collection which shed light on 
the so-called “Big Bang” theory of planetary devel- 
opment and background radiation. Within months, 
the COBE mission provided valuable data for 
numerous scientific papers and changed the text- 
books in astrophysics. 


Project Manager Roger Mattson introduces the infor- 
mative video with three challenges. First, COBE 
was by far the largest Goddard Space Flight Center 
in-house project to date. Secondly, COBE involved 
instrumentation at a level of extremely high sophis- 
tication, and the engineering challenge was great. 
Thirdly, the pressure was on after the Challenger dis- 
aster to achieve excellence. The Shuttle accident 
also meant COBE would be launched from a ELV 
rather than the Shuttle, and that the original budget 
would have to be expanded. 

COBE Project Scientist John Mather, who had con- 
ceived the COBE mission as early as 1974, notes that 
all scientific objectives were achieved or exceeded in 
measurement of a 15 billion-year-old phenomenon. 
COBE was launched nearly on time. 

Deputy Project Manager Dennis McCarthy explains 
how redesign for the Delta meant smaller volume 
and weight for the spacecraft, and that in turn mean 
rebuilding some disciplines at GSFC such as systems 
engineering and better contamination control. 

COBE Flight Assurance Manager Abigail Harper, 
who came on during the final year of the 10-year 
project, applauded the extensive reporting and docu- 
mentation on the project. She advised that 
Performance Assurance is accomplished best by 
visual inspection on the floor as well as analysis of 
documentation. 

Earle Young, COBE Instrument Manager, describes 
new procedures for contamination control and was 
among those who noted difficulty with the matrix 
organization, which is better for the institution than 
the project, and which responds technically, but not 
administratively, well. 

Roger Mattson explains the solution: a Skunk Works 
operation for the three dozen engineers who had to 
redesign for a Delta launch in one big room. GSFC 
had no such room, so eight trailers were hitched 
together, later becoming home for about the same 
number of Integration and Testing specialists. 
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The Skunk Works factory later became the COBE 
“War Room” where each subsystem schedule, man- 
ager’s name and action item was posted on the wall 
for all to see. 

Observatory Manager Anthony Fragomeni notes that 
the Skunk Works concept led to better control of 
money for procurement orders. He said the large 
success of COBE was “team spirit” engendered by 
the synergy of young and old on the job. 

John Wolfgang, COBE Integration and Testing 
Manager, said that despite a tight schedule and 
resources, it is so important to “do it right” and not 
cut comers on testing and analysis. Training and 
mentoring were considered vital as well. 


In sum. Project Manager Roger Mattson points to 
three major lessons learned. First, establish ground 
rules up front, with rigorous WBS and SOWs. 
Second, communications systems, internal and 
external were extensive. An open door policy led to 
monthly reporting systems, an electronics status 
report weekly and daily telecon with program man- 
agers at headquarters to cut off surprises. Third, 
technical testing procedures on the ground led to few 
engineering problems to be solved on orbit. 

Bendix Field Engineers provided technical assis- 
tance to this production for the NASA Program and 
Project Management Initiative. 
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